Setup up a Bitcoin Lightning point of sale in Common Lisp
Greek lambda letter shaped like a lightning by Midjourney
Because I can, I recently added support for testnet Bitcoin Lightning payments into this website. The beauty of Freedom and privacy respecting software is that you can do anything you want with it. There are many ways to do it. You can always use a service, provider for that, but the beauty of Bitcoin, the reason it is valuable, is that you can do it yourself.
Yes, there are common tools, the most established would be to use BTCPay. For my goals, it was an overkill, I didn’t want their entire infrastructure requirements, I didn’t want to work on C# and I really wanted to try Common Lisp for this project.
This project was more a pain than a delight. That is the nature of learning, you need inefficiencies and time waste to learn. I at least learn by mistakes, mistakes force me to pay attention. When things work, it does feel great, but I don’t learn anything new. I want to share this non trivial example, I always find the web full of toy examples that leave me wanting. That is why, when I find a project with a closed and manageable scope that was a pain for me to solve, I write about them , for you to learn from my mistakes.
Why was it a pain?
It is not to long ago that I learned Lisp, and I really love it. I wish I had learned about it earlier in my life. It is so flexible and expressive and simple. Its interactive developing process has no competition and today it is the language I use to try new ideas. Unfortunately that doesn’t mean it is the language I can use to do anything I want.
The language isn’t everything, there is also tooling, libraries, documentation and if I ever manage to become more social to take advantage of it: the community. I really like Common Lisp, the language, I feel the tools I managed to use are great. Libraries seem of high quality, yet they are really hard to find and there isn’t one for everything. Documentation is extremely hard to find. My well developed work mode of search based development is useless. There isn’t enough reachable content on the Internet to figure out how to do the things I want to do.
Yet, that wasn’t the main pain of this project. My pain was the front-end, how to deal with the user interface. Thus, it wasn’t Common Lisp’s fault, it was the lack of a good tool for that part of the job. It wasn’t any better than doing it in JavaScript, which nobody does this days. For a user interface, you need some kind of framework, and I couldn’t really find something for it. This is the unfair comparison, but ClojureScript+Reagent+Reframe are the only combination I have tried, that makes working on a front-end a manageable experience. It is not the language, it is the tools & libraries that really make the difference and I couldn’t find a similar experience in Common Lisp land for this project.
If you can’t find a good tool for the small job, don’t go big! Look how to do the minimum amount of work to get it done, without committing to a technology.
Front-end and Back-end total around 250 lines of code. The Back-end was a delight to write in Common Lisp, it beats any Python experience I have. There are good enough libraries to address my needs.
The Front-end a pain. I used Parenscript, which is a Lisp flavored JavaScript. That means, you are most of the time still in the JavaScript land of pain, and without a library to manage the User Interface any endeavor you take feels hopeless.
What does my solution look like?
The backend is a set of services, the frontend is a form and some JavaScript for the interactive parts. It is amazing that the Bitcoin & lightning networks require so little hardware to function and yet they let you to partake in a global trade network. Thanks to their design decisions around management of local state, keeping global consensus to the minimum, it is very possible for every individual to run your infrastructure and verify.
The next figure shows how little you need for this project. I use Bitcoin Core to connect to the Bitcoin network and participate on the global consensus. On top of that the LND daemon manages my wallet and manages my lightning channels, it too connects to the lightning network. LND exposes a REST API which is what my payment backend uses to request lightning invoices and listens to until the payment settles. That means the Payment Backend acts as a middle-ware exposing those same 2 API endpoints to the frontend user interface.
I’ll explain the system following the Lisp interactive development way. That means, building little by little each peace, refactor it then finally package everything. Simply summarizing the final code misses the advantage of doing it in Lisp.
I’ll jump between frontend and backend development. Interactive development for me starts with wishful thinking. Visualizing what you want and starting there, then continue to make it work.
Front-end: The user needs a form to request an invoice
The way payments in the Lightning Network work is by offering the user an invoice. That invoice tells the user’s lightning client where to pay and how much. I want to make it possible for the readers of this blog to endorse/reward my work on this blog by sending me a lightning payment for an amount they choose and with a message for which post earned their appreciation.
Thus take your favorite path for web development and craft a form with 2 fields,
one for text, one numeric and a submit button. I personally develop directly the
html and use on Emacs the impatient-mode to refresh the webpage as I type my
changes in. I do this in isolation for speed when developing individual
components. My general preference is to open an .html file load it with
tachyons and fontawesome from their CDN and live code the html.
<link rel="stylesheet" href="https://unpkg.com/[email protected]/css/tachyons.min.css"/>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.3.0/css/all.min.css" />
My resulting form, which you can see at the end of every post on this blog, is this simple html code.
<div id="app" class="pa4 black-80">
<div class="mw7 center br3 ba b--black-20 pa4" id="ln-payment">
<fieldset class="ph0 mh0 bn">
<legend class="f4 fw6 ph0 mh0">
Tip me with <i class="fas fa-flask green"></i>Testnet
<i class="fas fa-bolt gold"></i>Lightning
<i class="fab fa-bitcoin green"></i>Bitcoin
</legend>
<div class="mt3">
<label class="db fw6 lh-copy f6">Message:</label>
<input class="input pa2 input-reset ba bg-transparent w-100"
maxlength="64"
id="memo" type="text" value="Tip for this article">
</div>
<div class="mt3">
<label class="db fw6 lh-copy f6"><small>test</small> Sats:</label>
<input class="pa2 input-reset ba bg-transparent w-100"
id="sat-value" type="number" value="2345" min="2121" max="400000">
</div>
</fieldset>
<input class="b mt3 ph3 pv2 input-reset ba b--black bg-transparent grow pointer f6 dib"
id="get-invoice" onclick="javascript:invoice()" type="submit" value="Get lightning invoice">
</div>
</div>
This is great, because I can then use this same code as a partial template in hugo which powers this blog. I did suffer to get the styling right, because CSS is a pain, no matter how much tachyons has helped me to be declarative in styling, I have not yet a trained mind for visual design. But secondly, because this blog uses different CSS stylesheets and I had to deal with the conflicts.
Front-end: Reacting to user input, the JavaScript problem
The previous HTML code is the form, and the first sign for interactivity for the user is the onclick=“javascript:invoice()” on the submit button. That is where the nightmare starts. No matter how much I appreciate simple tools, few dependencies, managing the user interaction with the bare language is horrible.
The challenge was to do it in Common Lisp. Parenscript is the Common Lisp answer
for the JavaScript problem. It gives you a Lisp flavored JavaScript. I do find
it better than JavaScript, but not better enough, you only get to program in a
bit more expressive JavaScript. Secondly, the language is not enough for
managing a user interface, using the bare language is akin to a bare hands fight
in warfare. It is not possible to win. Yet, because this is a small self-hosted
project I stuck to it. I didn’t find the React wrapper documented enough to
venture that alternative, neither were other options interesting enough.
Documentation, examples, tutorials barely exist in this space. With this
experience I know to stick to ClojureScript+Reagent+Reframe for any relevant
project in my future.
Rant over, what is my solution? Open a .lisp file and start hacking the front
end client. I use Emacs as my editor, I use doomemacs as main configuration
manager for it. Doomemacs uses sly to work with Common Lisp code. As soon as
you open a .lisp file, it fires a Lisp repl for you to develop. I use SBCL as
my Common Lisp implementation, and I have Quicklisp for package management.
Quicklisp is wonderful as I can load new libraries while I develop. There is no
need to restart lisp and thus loose my program state at any time I need a new
library. That means I can discover, and try which dependencies I need to get my
system to work. For the interactive web development beyond design I drop
impatient-mode and move to skewer, and in this particular case with
Parenscript I use trident-mode to push my JavaScript changes into the browser,
without the need to manually refresh it. Because of doomemacs’ use of sly I
did have to patch trident-mode to use it, thus be warned. That is the price
and reward of using Free Software, you get the code to change it to your needs
and load it into your machine, but you have to do it yourself.
This is the starting code, I include the Emacs code comments to emphasize, that
tools are necessary for development, it is not simply code. The editor is your
assistant. In this case I declare this file using the lisp-mode which helps me
to interactively use the lisp repl and evaluate code. I also have a save hook
that evaluates the code in this file using trident-mode, that means it
translates this lisp code into JavaScript and loads it into the browser.
Instantaneous code reload, no need to refresh the page, no pain of loosing
state due to refresh.
;; -*- mode: lisp; -*-
;; evaluate this once to load parenscript
;; (ql:quickload :parenscript)
;; Evaluate this line once on the repl workaround so that output from trident doesn't get truncated.
;; (pushnew '(SLYNK:*STRING-ELISION-LENGTH*) slynk:*slynk-pprint-bindings* :test #'equal)
(defpackage ln-pay-js
(:use :cl :parenscript))
(in-package :ln-pay-js)
;; Local Variables:
;; eval: (add-hook 'after-save-hook #'trident-eval-buffer :append :local)
;; End:
Now let’s write effectful code. collect-invoice-data gets the value inside
the memo and value fields in the form and packs them into an object. You
immediately notice that no client side validation has taken place. A very bad
practice, but I have to cut corners to get this done. I will directly send the
strings of those fields to the backend, as you can recognize by the use of this
function inside invoice, which the user calls from the form.
(in-package :ln-pay-js)
(defvar *__PS_MV_REG*) ;; Work around parenscript multiple value return
(defvar +ln-api-uri+ "")
(defun collect-invoice-data ()
(create :value (chain document (get-element-by-id "sat-value") value)
:memo (chain document (get-element-by-id "memo") value)))
(defun invoice ()
(chain
(fetch (stringify +ln-api-uri+ "/make-invoice")
(create :method "POST"
:headers (create "Content-Type" "application/json")
:body (chain -J-S-O-N (stringify (collect-invoice-data)))))
(then (lambda (resp)
(if (eq 200 (@ resp status))
(chain resp (json))
(throw (@ resp status-text)))))))
As you see, this is JavaScript with Lisp syntax. I’m forced to use JavaScript
functions and put the parenthesis elsewhere. There is an advantage of structural
editing, which becomes ever less relevant with the use of tree-sitter. Yet,
even then I find lisp syntax quite useful. The stand alone function stringify
concatenates strings. +ln-api-uri+ is a variable containing the backend
server address. At the moment is the same development server and thus the empty
string, but that will change in production. create is to declare a JavaScript
object specified by key value paired arguments. -J-S-O-N is an abomination of
syntax, which translates to JSON in JavaScript, and here stringify has
actually a different meaning as it is the method of JSON. I’ll avoid this
pointless explanation of the code and let you see to what JavaScript it
translates to. It is an advantage of ParenScript that it produces readable
JavaScript, which is itself its limitation, as it can then only be a lisp
flavored JavaScript.
var __PS_MV_REG;
if ("undefined" === typeof LNAPIURI) {
var LNAPIURI = "";
}
function collectInvoiceData() {
return { 'value' : document.getElementById('sat-value').value,
'memo' : document.getElementById('memo').value };
};
function invoice() {
__PS_MV_REG = [];
return fetch([LNAPIURI, "/make-invoice"].join(""), {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(collectInvoiceData()),
}).then(function (resp) {
if (200 === resp.status) {
return resp.json();
} else {
throw resp.statusText;
}
});
}
__PS_MV_REG shows up to track multiple value returns. I don’t use that
beautiful lisp capability, that is why it remains empty, but I need to define
that variable for the JavaScript code not to crash. ParenScript doesn’t help
you with that.
Backend: Processing the invoice request
First load the dependencies into the lisp image. hunchentoot is the webserver,
data-format-validation helps to validate input, dexador is an http client to
query LND’s REST API, cl-json to work with JSON and finally arrows which
provides some syntactic sugar to “compose functions”.
Then define a package ln-pay and for the first development instances hard code
everything in variables. The reason I do this during development is because
using values directly is easy. If I need to load secrets, I can commit them to
code, because they are useless development secrets. The moment I need to move
to production, I voluntarily go over the pain of making each variable
configurable and consciously only create and properly account for the secrets I
do need. In this program I only need the authentication macaroon in a header and
where the LND REST endpoint is. Finally, I start the HTTP server, which I can
check works on http://localhost:4646. The :document-root option is a relative
path to where the .html files reside, start serving an index.html file
containing the earlier from.
(ql:quickload '(hunchentoot data-format-validation dexador cl-json arrows))
(defpackage ln-pay
(:use :cl :hunchentoot))
(in-package :ln-pay)
(defun encode-hex (bytes)
(with-output-to-string (o)
(loop for byte across bytes
do (format o "~2,'0x" byte))))
(defvar *auth-header*
(with-open-file (input "/path/to/lnd.macaroon"
:element-type '(unsigned-byte 8))
(let ((v (make-array 1024 :element-type '(unsigned-byte 8))))
(read-sequence v input)
(cons "Grpc-Metadata-macaroon" (encode-hex v)))))
(defvar *lnd-api-endpoint* "https://localhost:8481/v1")
(defvar *server* (start (make-instance 'easy-acceptor :port 4646
:document-root #p"www/")))
Now to the actual code to process the invoice request. The threading last macro
(->>) from arrows is useful for readability, each output of a function is
the input for the next as its last argument. The sequence of steps is easily
readable, it declares: Get the POST data, validate it is a non-nil string,
decode that string assuming it is JSON, then validate that JSON data structure,
because that is what we want to use to create the invoice. The valid data goes
into the request to LND and its result is finally returned.
(define-easy-handler (pay-req :uri "/make-invoice") ()
(setf (content-type*) "application/json")
(multiple-value-bind (response)
(arrows:->> (raw-post-data :force-text t)
(dfv:parse-input 'string)
(cl-json:decode-json-from-string)
(validate-invoice-request)
(request-invoice))
response))
The function for data validation is quite simple. I’m however unhappy that I specify my desired data inside the function, I would really prefer to specify a complete data model elsewhere, but for this simple example this works.
(defun validate-invoice-request (alist)
(flet ((entry (key) (cdr (assoc key alist))))
(list
:value (dfv:parse-input '(integer :min 1 :max 400000) (entry :value))
:memo (dfv:parse-input '(string :max-length 64) (entry :memo))
:private t
:expiry 600
:is_amp t)))
The request to LND is well encapsulated in the request-invoice function.
(defun request-invoice (content)
(dex:post (format nil "~a/invoices" *lnd-api-endpoint*)
:headers `(,*auth-header*
("Content-Type" . "application/json"))
:content (cl-json:encode-json-plist-to-string content)
:insecure t))
You might have noticed that at this point, all the functions only consider the happy path, where every input is good and produced the desired result. During prototyping, I don’t really care about exception handling. I deal with that as I develop.
Frontend: Updating DOM according to the response
Now that the frontend has a response, the sincere nightmare of dealing with a
user interface begins, things must change dynamically. The reply is JSON and
once into a JavaScript object, There is a single key I care about:
payment_request. It is a string I want to display and generate a QR code for
it too.
To generate the QR code, I use the library from Project Nayuki, which is simple and without dependencies. Yet, you must still implement the renderer. Following their examples, I wrote one in Parenscript to render on a HTML canvas.
(in-package :ln-pay-js)
(defun qr-canvas (payload canvas &key (scale 4) (border 0) (light-color "#FFFFFF") (dark-color "#000000"))
"Draws the QR Code on CANVAS with given PAYLOAD
Renders the CANVAS of width and height equal to (qr.size + BORDER * 2) * SCALE.
The drawn image is purely DARK-COLOR and LIGHT-COLOR, and fully opaque.
The SCALE must be a positive integer and the BORDER must be a non-negative integer."
(when (or (<= scale 0) (< border 0))
(throw (new (-range-error "Value out of range"))))
(let* ((ctx (chain canvas (get-context "2d")))
(qr (chain qrcodegen -qr-code
(encode-text payload (@ qrcodegen -qr-code -ecc "MEDIUM"))))
(width (* scale (+ (@ qr size) (* 2 border)))))
(setf (@ canvas width) width)
(setf (@ canvas height) width)
(loop for y from (- border) below (- width border) do
(loop for x from (- border) below (- width border) do
(setf (@ ctx fill-style) (if (chain qr (get-module x y)) dark-color light-color))
(chain ctx (fill-rect (* scale (+ x border))
(* scale (+ y border))
scale scale))))))
I want to place the QR code inside a modal, it should pop up and cover the screen. Design the modal again in isolation until you are happy with the design. Once the html is ready, I convert it into lisp form. The reason is I want to serve it directly from the JavaScript client.
(in-package :ln-pay-js)
(defun payment-modal-html ()
(who-ps-html
(:div :id "qr-send" :class "fixed top-0 left-0 w-100 h-100 flex items-center justify-center bg-black-80"
(:div :class "modal-background absolute top-0 left-0 w-100 h-100")
(:div :class "w-60-l relative"
(:div :class "flex items-center justify-between pa3 bb b--moon-gray bg-near-white br3 br--top"
(:legend :class "f4 fw6 ph0 mh0"
"Tip me with"
(:i :class "fas fa-flask green") "Testnet"
(:i :class "fas fa-bolt gold") "Lightning"
(:i :class "fab fa-bitcoin green") "Bitcoin")
(:button :id "close-modal" :class "f4 bg-transparent bn pointer" "x"))
(:div :id "invoice-body" :class "pa4 bg-white tc"
(:canvas :id "qr-canvas")
(:div :class "bg-moon-gray mv2 relative overflow-hidden br2"
(:div :class "progress-bar indeterminate h1 ma0 br3 bg-green"))
(:textarea :id "ln-req"
:class "mw9 w-100 mv2 h5"
"Generate an invoice first."))
(:div :class "flex pa3 bt b--moon-gray bg-near-white br3 br--bottom")))))
I also add to the CSS file the styling for an indeterminate progress bar.
.progress-bar {
background-image: linear-gradient(
-45deg,
rgba(255, 255, 255, 0.35) 25%,
rgba(255, 255, 255, 0) 25%,
rgba(255, 255, 255, 0) 50%,
rgba(255, 255, 255, 0.35) 50%,
rgba(255, 255, 255, 0.35) 75%,
rgba(255, 255, 255, 0) 75%,
rgba(255, 255, 255, 0)
);
background-repeat: repeat-x;
background-size: 2rem 2rem;
animation: stripes 2s linear infinite;
}
@keyframes stripes {
to {
background-position: 20px 0;
}
}
.progress-bar.indeterminate {
position: relative;
animation: progress-indeterminate 5s linear infinite;
}
@keyframes progress-indeterminate {
from {
left: -35%;
width: 35%;
}
to {
left: 100%;
width: 25%;
}
}
From this moment on I truly missed using ClojureScript and Reframe. Yet my
solution is as follow. Always clear all DOM state to have a clean workplace.
Use the function that generates the HTML text and inject it into an element
using the insert-adjacent-h-t-m-l function. Hook new event listeners to clear
elements that were just added. Inject into the modal the payment_request
string and render the QR code in the designated canvas node.
(in-package :ln-pay-js)
(defun clear-state-nodes ()
"Remove from DOM all nodes which represent state from previous request.
That is error boxes and modal containers of payment."
(chain document (query-selector-all ".invoice-error,#qr-send")
(for-each (lambda (x) (chain x (remove))))))
(defun payment-modal (node-id)
"Dress NODE-ID element with a payment modal and add corresponding close events"
(clear-state-nodes)
(chain document (get-element-by-id node-id)
(insert-adjacent-h-t-m-l "beforeend" (payment-modal-html)))
(chain document (query-selector-all ".modal-background,#close-modal")
(for-each (lambda (node) (chain node (add-event-listener "click" #'clear-state-nodes))))))
(defun load-qr (lnreq)
(payment-modal "app")
(let ((text-con (chain document (get-element-by-id "ln-req")))
(canvas (chain document (get-element-by-id "qr-canvas")))
(payment-request (@ lnreq payment_request)))
(setf (@ text-con value) payment-request)
(qr-canvas payment-request canvas)))
It is of course necessary to update the invoice function to handle the response from the backend and use these functions.
(in-package :ln-pay-js)
(defun invoice ()
(clear-state-nodes)
(chain
(fetch (stringify +ln-api-uri+ "/make-invoice")
(create :method "POST"
:headers (create "Content-Type" "application/json")
:body (chain -J-S-O-N (stringify (collect-invoice-data)))))
(then (lambda (resp)
(if (eq 200 (@ resp status))
(chain resp (json))
(throw (@ resp status-text)))))
(then #'load-qr)))
At this point, it works for us. The user sees the modal pop up with the invoice request. He could pay and close the modal himself. However, on our side, we only treated the happy path. Things could go wrong and handling those exceptions is necessary.
Handling errors
There are many things which can go wrong. Invalid User input, general bad input, the LND service could reply unexpected errors among other things. Let’s first handle those cases on the backend using a macro. Everything is taken care of at the same level, only changing the response message for the user. The operation either succeeds or fails as a bad request.
(defun err (message &rest args &key &allow-other-keys)
(setf (hunchentoot:return-code*) 400)
(cl-json:encode-json-plist-to-string
`(:error ,message ,@args)))
(defmacro handle-all-errors (&body body)
`(handler-case
(progn ,@body)
(dex:http-request-failed ()
(err "Communication to Lightning Network failed"))
(dfv:invalid-format (c)
(err (dfv:invalid-format-reason c)))
((or cl-json:json-syntax-error
cl-json:unencodable-value-error) ()
(err "Invalid JSON input"))
((or type-error cl-base64:base64-error) ()
(err "Invalid input"))
(error (e)
(format t "ERROR: UNKNOWN ~A~%" e)
(err "Unknown error"))))
(define-easy-handler (pay-req :uri "/make-invoice") ()
(setf (content-type*) "application/json")
(handle-all-errors
(multiple-value-bind (response)
(arrows:->> (raw-post-data :force-text t)
(dfv:parse-input 'string)
(cl-json:decode-json-from-string)
(validate-invoice-request)
(request-invoice))
response)))
On the front-end errors will appear inside error boxes instead of triggering the modal.
(in-package :ln-pay-js)
(defun signal-error (error-msg)
(chain document (get-element-by-id "ln-payment")
(insert-adjacent-h-t-m-l
"beforeend"
(who-ps-html
(:div :class "invoice-error bg-washed-red ba br2 ma3 pa3 red"
(:div :class "message-body" error-msg))))))
(defun invoice ()
(clear-state-nodes)
(chain
(fetch (stringify +ln-api-uri+ "/make-invoice")
(create :method "POST"
:headers (create "Content-Type" "application/json")
:body (chain -J-S-O-N (stringify (collect-invoice-data)))))
(then (lambda (resp)
(let ((parsed (chain resp (json))))
(if (@ resp ok)
parsed
(chain parsed (then (lambda (json)
(throw (@ json error)))))))))
(then #'load-qr)
(catch #'signal-error)))
Listening to payment
Once error handling is properly set and the user is properly warned about errors, it is time to add the last convenience for the user. After he pays, the user interface must update. The waiting modal changes to thank the user for the payment and then closes.
The correct way would be to subscribe to LND for payment updates, and keep the
user subscribed to the server for it to know when the payment has been
registered. That is however more problems than I want to solve today. The easy
solution is, with a given timeout, query the backend for the payment status. The
backend in turn queries LND very frequently and as soon as the payment has
settled it replies to the client. If the timeout is reached and the payment is
not settled the backend still replies, yet with an OPEN state. On the client,
this re-triggers a request, and so it keeps waiting for the update. Would the
back end reply CANCELLED (invoice expired), then the modal changes to notify
the user about that too.
This method saves requests between client and backend, and loads the backend. Because of that asymmetry it exposes the backed to unnecessary load. For the first iteration and as long as readers of this blog aren’t maliciously creating requests to my server, this is good enough. I monitor my server to track its health, and I have enough time to implement the correct solution latter on, should I have a lot of traffic, which is anyway a nice problem to have.
My solution on the front end is:
(in-package :ln-pay-js)
(defun clear-invoice-modal (success-p)
(setf (inner-html (chain document (get-element-by-id "invoice-body")))
(if success-p
(who-ps-html
(:div :class "center bg-washed-green green w-90 ma2 f4 br3"
(:div :class "pa3 bb b--moon-gray bg-green br3 br--top white fw6"
"Payment Completed!")
(:div :class "pa3"
"Thank you for your kind support!")))
(who-ps-html
(:div :class "center bg-washed-red red w-90 ma2 f4 br3"
(:div :class "pa3 bb b--moon-gray bg-red br3 br--top white fw6"
"Invoice has expired!")
(:div :class "pa3"
"Please request a new one.")))))
(set-timeout #'clear-state-nodes (if success-p 1500 5000)))
(defun json-or-error (response)
(let ((parsed (chain response (json))))
(if (@ response ok)
parsed
(chain parsed (then (lambda (json)
(throw (@ json error))))))))
(defun listen-for-payment (lnreq)
(chain
(fetch (stringify +ln-api-uri+ "/listen-payment")
(create :method "POST"
:headers (create "Content-Type" "plain/text")
:body (@ lnreq r_hash)))
(then #'json-or-error)
(then (lambda (resp)
(let ((state (@ resp :state)))
(cond
((equal "CANCELED" state)
(clear-invoice-modal nil))
((equal "SETTLED" state)
(clear-invoice-modal t))
((listen-for-payment lnreq))))))
(catch (lambda (error)
(clear-state-nodes)
(signal-error error)))))
(defun invoice ()
(clear-state-nodes)
(chain
(fetch (stringify +ln-api-uri+ "/make-invoice")
(create :method "POST"
:headers (create "Content-Type" "application/json")
:body (chain -J-S-O-N (stringify (collect-invoice-data)))))
(then #'json-or-error)
(then (lambda (response)
(load-qr response)
(listen-for-payment response)))
(catch #'signal-error)))
Notice that once again I extended invoice to now trigger the
listen-for-payment event.
The back end is intentionally simple:
(defun invoice-state (hex-rhash)
(arrows:->>
(dex:get (format nil "~a/invoice/~a" *lnd-api-endpoint* hex-rhash)
:headers `(,*auth-header*
("Content-Type" . "application/json"))
:insecure t)
(cl-json:decode-json-from-string)
(assoc :state) (cdr)))
(defun payment-state (hex-rhash &optional (repeat 30) (sleep 2))
(do ((request 1 (1+ request))
(state (invoice-state hex-rhash) (invoice-state hex-rhash)))
((or (member state '("SETTLED" "CANCELED") :test #'equal)
(> request repeat)) state)
(sleep sleep)))
(define-easy-handler (listen-payment :uri "/listen-payment") ()
(setf (content-type*) "application/json")
(handle-all-errors
(multiple-value-bind (response)
(arrows:->> (raw-post-data :force-text t)
(dfv:parse-input 'string)
(cl-base64:base64-string-to-usb8-array)
(encode-hex)
(payment-state)
(list :state)
(cl-json:encode-json-plist-to-string))
response)))
Doing it again: going to production
Now that all the moving pieces are in place, and everything works on my machine it is time to do it again. Review the entire code, change all hard coded values for environment variables. Refactor, create an executable, setup a service, include CORS headers because the payment processor is a different server than my blog server.
Serving the JavaScript is certainly the ugliest. During development I dynamically
pushed JavaScript using the trident-mode, yet it is useful to serve it, because
I do refresh from time to time the page. In the production environment, I use it
in different ways. I have manually copied the compiled file to my blog, and on top
of that I serve a frozen version of the file, with its server configuration.
(defun serve-js (env)
(if (equal env "DEV")
(push
(hunchentoot:create-regex-dispatcher
"/invoices\.js"
(lambda ()
(setf (hunchentoot:content-type*) "text/javascript")
(ps:ps-compile-file
(asdf:system-relative-pathname "ln-pay" "src/invoices.paren"))))
hunchentoot:*dispatch-table*)
(let* ((js-code #.(ps:ps-compile-file
(asdf:system-relative-pathname "ln-pay" "src/invoices.paren")))
(api-uri (format nil "~a:~d" (uiop:getenv "LN_PAY_HOST")
(uiop:getenv "LN_PAY_PORT")))
(conf-code (ps:ps (setf +ln-api-uri+ (ps:lisp api-uri))))
(out-file (concatenate 'string js-code conf-code)))
(push
(hunchentoot:create-regex-dispatcher
"/invoices\.js"
(lambda ()
(setf (hunchentoot:content-type*) "text/javascript")
out-file))
hunchentoot:*dispatch-table*))))
Starting the server has now more steps, all the hard coded variables are now queried from environment variables. I also support SSL on the production server. At the moment, I dare to expose the service to the wild, wild internet. That might change in the future and I’ll offer the services through a reverse proxy.
(defvar *auth-header*)
(defvar *lnd-api-endpoint*)
(defvar *server*)
(defun start-server (&key port)
(setf *auth-header*
(with-open-file (input (uiop:getenv "LND_MACAROON_PATH")
:element-type '(unsigned-byte 8))
(let ((v (make-array 1024 :element-type '(unsigned-byte 8))))
(read-sequence v input)
(cons "Grpc-Metadata-macaroon" (encode-hex v))))
*lnd-api-endpoint* (format nil "~a/v1" (uiop:getenv "LND_SERVER")))
(let ((host (quri:uri (uiop:getenv "LN_PAY_HOST"))))
(setf *ac-origin* (format nil "~a://~a" (quri:uri-scheme host) (quri:uri-domain host))))
(setf *server* (if (equal (uiop:getenv "ENV") "DEV")
(make-instance 'hunchentoot:easy-acceptor
:port port
:document-root (asdf:system-relative-pathname "ln-pay" "www/"))
(make-instance 'hunchentoot:easy-ssl-acceptor
:port port
:document-root (asdf:system-relative-pathname "ln-pay" "www/")
:ssl-certificate-file (uiop:getenv "SSL_CERT")
:ssl-privatekey-file (uiop:getenv "SSL_KEY"))))
(serve-js (uiop:getenv "ENV"))
(hunchentoot:start *server*))
Because my blog and my payment processor service exist on separate domains I must configure CORS. I can solve this with a macro that wraps the definition of my API endpoints. Set the OPTIONS request for the preflight to respond with empty string. The else clause in the conditional is for the POST request, which is the actual request.
(defvar *ac-origin*)
(defmacro cors-handler (description lambda-list &body body)
`(define-easy-handler ,description ,lambda-list
(setf (header-out "Access-Control-Allow-Origin") *ac-origin*)
(setf (header-out "Access-Control-Allow-Methods") "POST,OPTIONS")
(setf (header-out "Access-Control-Max-Age") 86400)
(setf (header-out "Access-Control-Allow-Headers") "Content-Type")
(setf (header-out "Content-Type") "application/json")
(setf (content-type*) "application/json")
(if (eq :options (hunchentoot:request-method*))
""
(handle-all-errors ,@body))))
(cors-handler (pay-req :uri "/make-invoice") ()
(multiple-value-bind (response)
(arrows:->> (raw-post-data :force-text t)
(dfv:parse-input 'string)
(cl-json:decode-json-from-string)
(validate-invoice-request)
(request-invoice))
response))
(cors-handler (listen-payment :uri "/listen-payment") ()
(arrows:->> (raw-post-data :force-text t)
(dfv:parse-input 'string)
(cl-base64:base64-string-to-usb8-array)
(encode-hex)
(payment-state)
(list :state)
(cl-json:encode-json-plist-to-string)))
To run a standalone binary, you must incorporate an entry point. The main
function starts everything, joins the webserver thread to the foreground,
otherwise the executable will finish.
I also sneak a slynk server. This is the beauty of LISP. I can even after
deployment connect to the running image and make changes to it if I need to.
That is not an open connection to the wild internet, I must go over SSH to the
server and then locally connect to it.
Once the webserver is on the main thread, it needs to listen to interrups to kill the server.
(defun main ()
(start-server :port (parse-integer (uiop:getenv "LN_PAY_PORT")))
(format *standard-output* "Hunchentoot server started.~&")
(slynk:create-server :port (parse-integer (uiop:getenv "SLYNK_PORT")) :dont-close t)
(handler-case (bt:join-thread (find-if (lambda (th)
(search "hunchentoot" (bt:thread-name th)))
(bt:all-threads)))
;; Catch a user's C-c
(#+sbcl sb-sys:interactive-interrupt
#+ccl ccl:interrupt-signal-condition
#+clisp system::simple-interrupt-condition
#+ecl ext:interactive-interrupt
#+allegro excl:interrupt-signal
() (progn
(format *error-output* "Aborting.~&")
(hunchentoot:stop *server*)
(uiop:quit)))
(error (c) (format t "Woops, an unknown error occured:~&~a~&" c))))
Conclusion
You made it till the end, I’m glad you find this an interesting read. While learning Common Lisp I constantly struggled to find resources, this is my contribution to improve that.
After everything is done, it doesn’t look too bad. The code looks simple enough for a simple problem. The interactive programming model of LISP is incredibly helpful. Common Lisp beets Python, even as a beginner in Lisp and quite experienced in Python I already dare to say there is nothing Python as a Language does better. As an ecosystem, Python has every imaginable library, some domains like Data Science are even dominated by Python. There just a massive amount of work put into the Python ecosystem which you can leverage. That means, unless I’m really working on a complete novel algorithm I would prefer Python to Common Lisp. It is not the language is the libraries.
The true pain for me in this project was the JavaScript problem, and Python would have not been able to help me here either. Trying to keep the User Interface manageable from a software point of view is a hard problem, that is why so many JavaScript frameworks and tools exist. More interactions, immediate client side data validation, managing state, are necessary for any decent application and I skipped over them in this project. All those things feel incredibly hard when all you have is the bare JavaScript(ParenScript) language. Doing a Web Application without a Framework that helps you manage that complexity is a lost battle. From my experience ClojureScript+Reagent+Reframe is the most convenient developer experience, I fail to see why that hasn’t taken over the world.
Common Lisp, despite all its language quirks feels to me better than Clojure.
Just by the fact that starting the SBCL image and loading new libraries at
runtime is possible and fast, the developer experience is so much better.
Clojure is just getting there. On top of that Sly, which is only an aggressive
improvement over slime, which has years of development, is just better tooling
to interact with the lisp image. CIDER has still a lot to catch up. They
probably will, because the Clojure community is bigger. Until then, I’ll
probably will keep jumping between Clojure and Common Lisp.
Maybe there is a hidden gem in the Common Lisp world to develop web clients, I personally coudn’t find it. Yes, there are some attempts at that, but they neither fit my mental model nor those models I already know. Yes, I’ll always want to learn new ones, but mentoring documentation is really hard to find in the land of Common Lisp all that you have is the HyperSpec, which is not beginner friendly. You can’t expect to change someones mental models without writing beginner friendly tutorials.
Dr. Óscar Nájera
Software distiller & Recovering Physicists
As scientist I studied the very small quantum world. As a hacker I distill code, because software is eating the world, and less code means less errors, less problems, more world.