Building a SaaS product with Clojure

The Idea

Like many software developers, I spend a lot of my spare time on side projects.  Some side projects are purely for my own edification: learning a new language (such as my current favorite, Clojure), or  exploring a new technology or problem domain.  However, a great many project ideas that tend to pop into my head at random intervals tend more toward the entrepreneurial.  These sorts of ideas either originate internally (usually the most insane or impractical, since I tend to live inside my own head a little too much), or as a result of observing (or experiencing) problems for which a good solution is not immediately evident.  The former can usually be disposed of quickly by subjecting them to a few thought experiments, while the latter type tend to be more persistent, begging to be investigated further.

So, enter my current obsession with creating an enterprise wiki that isn’t painful to use.  I’ve used many expensive “Knowledge Management” and corporate wiki tools over my career.  Some are better than others, but I have yet to find one that I actually like.  Either they have serious problems in the information search and discover-ability department (e.g., SharePoint), or they seem to be designed around the idea that you want to spend your time interacting with the application itself rather than finding the information you need before going back to your actual job-related tasks (e.g., various tools with overwrought interfaces with hundreds of knobs and buttons that consume half of the available screen real estate).

I want a KM tool in which content is front-and-center and an interface that stays out of the way.  I also want a functional search capability and a way to browse information and content when I don’t already know exactly what I’m looking for.   My first attempt at this ended up in what was basically a traditional wiki with perhaps a nicer interface.  The voices in my head laughed at me, saying “So, what’s different about this thing?  You’ve created nothing new.  Forget it and try again.”  Eventually a new concept presented itself to me, which I’ve named Contabulo.  A wiki with a much more visual interface – inspired by card-based PM tools such as Trello, but geared toward being a knowledge management and collaboration tool.  The basic idea for the interface came to me when I observed my wife using Pinterest: “There should be a productivity tool that looks kind of like that.”  Earlier versions of Contabulo even used a masonry-like layout, but I quickly realized that was a poor fit and switched to a gridded layout based on CSS-grid (which happened to gain majority browser support just in time to be usable).

Given that I was tackling this project alone, in my spare time, I needed to select tools and technologies that would help me maximize my personal productivity.  While I’m paid good money to develop in Java at my day job, I have to say Java is absolutely atrocious for individual productivity.  I could write at length about my problems with Java, but suffice to say that a high-ceremony language that forces you into designing your software architecture in terms of class hierarchies (i.e., OOD) does a pretty good job of sucking the joy (and speed) out of software development.  When I’m beginning a new personal project where I have total freedom of choice, I will generally gravitate toward a functional programming language.  Being that I’m both a huge Lisp fan and also very (albeit somewhat unwillingly) familiar with the Java ecosystem, I chose Clojure to implement the backend services.

Architecture

My background in enterprise software development is clearly evident in the system architecture 🙂  My favorite relational database, PostgreSQL, serves as the primary data-store.  Most of the CRUD (Create, Read, Update, Delete) functionality is taken care of by the “API Service”, which is built around the excellent compojure-api.  Now, when it comes to working with a relational database, I’m generally used to being forced to do things the Java way and use an ORM like hibernate or eclipselink.  However, with a functional programming language an ORM doesn’t make sense (not that an ORM makes sense in any case, but I digress), and I found HugSQL to be an awesome way to work with a SQL database.  I’ll take writing SQL queries anytime over composing JPA/Hibernate criteria queries.  Additionally, rows are written to and read from the database as simple Clojure maps and vectors, rather than having to define domain objects for every table in the database.

The contabulo-auth service is responsible for verifying user login credentials an issuing tokens.  Tokens are generated and verified using the buddy-sign library.

Elasticsearch powers the search functionality, provided by the “Search Service” and is kept in sync with the main database via an “Index Service.”  I actually opted to use Elastic’s own Java SDK for interacting with elasticsearch, writing Clojure wrappings as necessary (One of the many benefits of running on the JVM is the wealth of pre-existing Java libraries for just about anything you can think of — the only downside being they’re written in Java and you’ll have to wrap them in your language of choice).

Synchronous communication between processes is accomplished using HTTP RESTful interfaces, while asynchronous tasks are dispatched via RabbitMQ.  The contabulo-mail service, for example, is responsible for sending email notifications to users and receives all of its tasks via a JMS queue.  For communicating with RabbitMQ from Clojure services, the Langohr library is used.

The frontend is a rich javascript web application (I decided to stick with plain javascript/ES6 rather than go with clojurescript) written using the mithril.js framework.  Mithril is a great little client-side framework that boasts a very functional-style of javascript programming (truth be told, you can be quite functional with javascript, but it requires a lot of discipline).

Clojure (and FP) Benefits

Thanks to Clojure, I have backend services composed from side-effect free functions and immutable data structures.  Whole classes of runtime errors I’m used to seeing on large Java applications are simply absent from Contabulo.  This makes iterating on the product and adding new features *so* much easier.   Further, ‘statelessness’ is considerably easier to achieve in Clojure, partly due to it’s non OO-ness, partly due to its immutability-by-default design.  This means that adding additional instances of each service will be much easier.

I’ve also found the unit testing experience in Clojure to be vastly superior to  Java’s.  Though, I doubt this is due to any inherit superiority of clojure-test over JUnit but rather simply a result of the fact that side-effect free deterministic functions are easier to unit test.  In any case, it’s still a win.  As an aside, I’ve also found Scala to have a much more pleasant testing experience than Java (probably tied with Clojure in that regard).

While the frontend web application was written in ES6, the mithril.js framework’s functional style has resulted in a rich web application constructed from modular, de-coupled components that can be reused easily.  This makes regressions on the frontend much less likely as well (Which is good in my case, because I’m really much more of a backend developer).

I will have to admit to writing one small transient service in a non-FP language.  A small program written in Golang is triggered by a systemd timer periodically to perform housekeeping tasks (moving files around, doing some basic database ops, etc).  This was one case where I didn’t want to suffer the overhead of spinning up a JVM every ten minutes and since the housekeeping service does nothing but I/O operations (which tend to be inherently procedural anyway), I decided to go (pun sort of intended) with something simple.

Conclusion

I’ve heard some critics say that Clojure is unsuitable for big projects with large teams, usually due to some combination of dynamic typing and  difficulty in finding Clojure developers.  I’m not sure I’d agree with that assessment.  I think prodigious use of type hints and tools like schema (and more recently, clojure.spec) can largely mitigate most of the problems caused by dynamic typing, and I’m not convinced the latter issue even exists.  Contabulo is the largest project I’ve done using Clojure (or any Lisp, for that matter) and now that the all of the basic architecture and core functionality is in place, I’m quite pleased with how quickly I was able to build it and how easy it is to add new features or make modifications.  I think that’s a win.  Now it remains to be seen how Contabulo succeeds as a product.