On Boards and Cards

Unless you’ve been living under a rock, you’ve doubtless noticed the current trend in project management tools involving “boards full of cards.”  Trello is probably the most well known (and cheapest, being free, which might have something to do with its popularity) of this archetype.  Other, more elaborate (and expensive) enterprise tools have implemented “agile” boards as part of their offerings.

Personally, I think most of these tools are well suited for their intended use case:  Agile project management.  However, as someone who’s tried to use Trello as a more, erm, general purpose tool, I’ve found it lacking (shopping lists? No.  Notes? No).  Which is fine, it’s a specialized tool designed for a particular set of use cases, which just so happen to not align with my needs.  Most of these PM tools assume the following:

1.)  You are managing projects (duh).  Which are temporary efforts with a definite start and end date.  Thus, boards are meant to be used temporarily.

2.)  You’re using some sort of Agile (or perhaps Kanban) Project Management methodology.  You may not like Agile, or it may not suit your particular effort.

3.) They’re not geared toward long term knowledge management, you’ll need a completely separate tool (such as a wiki) for this.

Issues 1 and 3 are related.  You want to store lessons learned, institutional knowledge and other “knowledge” artifacts created during the course of a project.  The traditional answer to this is to make your engineers enter this knowledge into a wiki, as the project board that was so nice for organizing and tracking tasking turns out not to be so great when it comes to organizing and archiving information for long term usage.  This, of course, introduces yet another ‘chore’ for engineers who would rather be building things.  And it boils down to the fact that taking the time to enter information into a wiki takes people out of their normal workflow.  Thus, it is frequently forgotten and/or skipped when deadlines loom.  I, for one, have never seen a corporate wiki that was well maintained and kept up-to-date 🙂

So, I think a worthy goal should be to keep information together as much as possible.  Place notes, design artifacts, project planning documents, meeting minutes, miscellaneous documentation, etc. in the same system you also place your tasking “cards,” thus folding “updating the wiki” into your normal workflow.  The trick, of course, is to keep this vast pile of information from turning into an unmanageable, unsearchable, and unusable mess.

That’s the idea I’ve been pursuing with my latest project, which I’ve named Contabulo.

Half wiki, half board

contabulo_screenshot1

An example Contabulo project board

From the screenshot, at first blush, Contabulo looks similar to an agile board.  Unlike Trello and friends, however, these cards aren’t organized into lists.  In fact, there’s not a set ordering or arrangement at all as the interface assumes (at least on a well-populated board) that you’re going to be making use of that search bar to find the cards you seek.  The search feature is powered by Elasticsearch on the backend, so it does a fairly decent job (the ‘main’ database is PostgreSQL).  As you can see from the graphic, cards can be assigned different colors, and you can assign a board background image, which I know is pure vanity.

Cards

contabulo_donut_card

A Contabulo card with a header image

Contabulo cards are designed to be flexible, and contain anything from a quick note to an entire ‘wiki’-esque article.  The editing format is Markdown (specifically, Commonmark with a few extensions), and I don’t know about you, but I much prefer Markdown to the Wikitext format you’ll find in MediaWiki based wikis 🙂  As you can see from the image above, you can also attach files (including images), and can specify a card image to appear in the header.

Card_in_edit_mode

A card being edited using Markdown – at some point I’ll add a rich editor

In addition to the card body and attachments, you can add special ‘content’ blocks to a card.  As of this writing, checklists are supported, but it is easy to imagine adding many other types of useful content: maps (think – directions to events), discussion/comment sections, integrations with third party apps, etc.  You can also add ‘tags’ to cards, which of course are optional, but the search index does use to improve results.

Groups

Contabulo has the concept of ‘groups’ in order to control access to different boards.  Each user can have ‘personal’ boards which are either totally private, or can be given public read-only access for outsiders.  “Group” boards can likewise be visible only to the group, or additionally be read-only to non-group members (one notional use of a public read-only board could be a job board).  Each group member can be assigned read-only, editor, or admin-level access.

group_screen

Managing groups in Contabulo

Tech

For the curious, I’ll give a brief overview of the Contabulo “tech stack” (I think that’s what the kids are calling it these days).  I’ve been building Contabulo in my spare time (evenings and weekends), since right now I have to keep a day job in order to eat and pay the mortgage.  Thus, the need to maximize my personal productivity was definitely a major factor in selecting tools and technologies.  The backend is written primarily in Clojure, which is in the Lisp family of languages and runs on the Java Virtual Machine.  I chose Clojure because:

  1. I Like Lisp, and think it’s an amazingly productive language
  2. I have a large amount of familiarity with the JVM and the Java ecosystem in general (and there are a *huge* number of helpful Java libraries which you can easily call from Clojure), even if I may not care much for the Java language itself 🙂

I briefly flirted with the idea of writing the backend services in Haskell after working through Learn You a Haskell (great book, btw), but then the fever broke and I got better.

So, I’ve split up my backend into several Clojure services (not quite as fine-grained as ‘microservices‘ would be, but I don’t care), and one little database maintenance/housekeeping utility I wrote in Golang that runs once an hour so.  interprocess communication is largely using RabbitMQ, though some services call each other using HTTP when synchronous communication is needed.  I’m using my favorite webserver, Nginx, as a load-balancer (and also for SSL termination), and all of my backend services are stateless so that running multiple instances of each shouldn’t be a problem.  I’ve tried to lay the foundation for a scalabe architecture which will hopefully prove necessary.  I suppose I could have probably just spun up a Ruby on Rails app much more quickly, but where is the fun in that? 🙂

As I mentioned earlier, the main database is PostgreSQL, and search is powered by Elasticsearch (though a Clojure ‘search’ service sits in front of the Elasticsearch service, since ES, as with any database, shouldn’t be directly exposed to the Internet).

For the web application, I used ES6 with the Mithril microframework and Bootstrap, along with a plethora of smaller Javascript libraries, of course. I’m not the most experienced or skilled frontend developer in the world, but I think I do ok (when properly incentivized, like when there’s noone else to do it).

Conclusion

Contabulo isn’t my first attempt at a Software-as-a-Service (SaaS) offering, but I feel pretty good about this one.  I find it useful, in any case, and will continue to use it (and keep it updated) for my own needs.  My focus right now is to complete testing and squash any show-stopping bugs so I can launch this thing and hopefully start getting some paying users.  I have a long list of features I want to implement, but I think Contabulo is far enough along it’s time to start getting user feedback in order to guide its future direction.  If you’d like to receive updates (i.e., to be notified when Contabulo is live), please sign up for the mailing list!

Implementing Search-as-you-type with Mithril.js

I’ve been working on a new project, the front-end of which I’m coding up in ES6 with Mithril.js (using 1.0.x now after spending the better part of a day migrating from 0.2.x).  I wanted to implement “search as you type” functionality, since I’m using Elasticsearch on the back-end for its full-text search capability.

search_box

A simple user search box

Took me a bit of trial-and-error, but I came up with this mithril component that provides a simple text field that automatically fires a callback periodically (the timing is configurable)

export let SearchInputComponent = {

    /**
     * attributes:
     * helpText - small text to display under search box
     * callback - callback to be fired when timeout/minchar conditions are met
     * minchars - min number of characters that must be present to begin searching
     * inputId  - DOM ID for input field
     * @param vnode
     */
    oninit: function(vnode) {
        vnode.attrs = vnode.attrs || {};
        // Search box caption
        vnode.attrs.helpText = vnode.attrs.helpText || "Enter search text here";
        //Callback to fire when user is done typing
        vnode.attrs.callback = vnode.attrs.callback || function(){};
        //Minimum number of characters that must be present to fire callback
        this.minchars = vnode.attrs.minchars || 3;
        this.timeout = vnode.attrs.timeout || 1e3; //default 1 second for input timeout
        this.inputId = vnode.attrs.inputId || "searchTextInput";
        this.timeref = null;
        this.err = stream(false);

        this.doneTyping = function(){
            let searchString = $('#' + this.inputId).val();
            if (this.timeref && searchString.length >= this.minchars){
                this.timeref = null;
                vnode.attrs.callback(searchString);
            }

        }.bind(this);

        this.oninput = function(event){
            if (this.timeref){
                clearTimeout(this.timeref);
            }
            this.timeref = setTimeout(this.doneTyping, this.timeout);
        }.bind(this);

    },

    view: function(vnode) {
        return m("fieldset", {class: "search-input-box"}, [
            m("input", {type: "text", class: "form-control", id: vnode.state.inputId,
                autofocus: true, "aria-describedby": "searchHelp", oninput: vnode.state.oninput,
                onblur: vnode.state.doneTyping}),
            m("small", {id:"searchHelp", class: "form-text text-muted"}, vnode.attrs.helpText)
        ]);
    }

};

Pretty basic.  In my case, I wanted to avoid firing my search callback (which makes a request to my back-end search service) until a certain number of characters had been entered.

CouchDB Authorization

For the last month or so I’ve been working on a little project using CouchDB and Go.  As this is my first time using a NoSQL database, I’ve had to learn to think a little bit differently about how I interact with my data.  One aspect of Couch that had me a little confused at first was how it deals with authentication and authorization.

When creating an application with a traditional RDBMS (with PostgreSQL being my favorite), I was used to handling authentication / authorization myself (usually via a third-party library) at the application level.  While you can of course still do this with CouchDB, when I first came across CouchDB’s auth support, I thought “Great! I can just use Couch for everything!”.  Turns out, I sorta can, but it’s not as simple as I had first assumed.  Oh well 🙂

Users are stored in a special database called “_users” and creating a new one is as simple as a PUT request (Admin credentials are generally required for creating or deleting users).  For authentication, you can either use Couch’s Session API, or in the case of HTTP Basic, just pass the client’s Basic Authentication headers through to Couch from your application. This was easy enough to implement and seems to work well enough for what I need.

Authorization, however, is where things can get tricky.  CouchDB handles permissions on a per database basis.  It is not possible to restrict access to individual documents.  “Well, that’s useless, guess I’ll have to handle authorization some other way” I thought to myself.  But the answer, if your use-case allows for it, is just to create multiple databases at the level of granularity you need to control access.  I resisted doing this at first, as being used to RDBMS systems, I kept thinking “a database is a big deal and creating dozens, hundreds, or even thousands of them is insane.”  But CouchDB is not Postgres 🙂 and a little research told me that creating a database per user account is actually a very common use case and doesn’t cause any serious issues (if you start getting into thousands of databases you may need to tweak your config to allow a larger number of databases to be open simultaneously).

CouchDB allows you to grant access to databases either on a per-user basis or based on user roles.  I would highly recommend doing role-based authentication as I think that’s more flexible.  To specify database users/roles, you edit a database’s security document (called ‘_security’).  The structure of the security object looks like this:

{
    "admins": {
        "names": [
            "superuser"
        ],
        "roles": [
            "admins"
        ]
    },
    "members": {
        "names": [
            "user1",
            "user2"
        ],
        "roles": [
            "developers"
        ]
    }
}

CouchDB allows two different levels of user: ‘members’ and ‘admins.’  Members can read and write documents in the database while admins can edit design documents as well as the database security document.  One thing to note here is that if no users/roles are specified, CouchDB will default a database to ‘public’ access, so… definitely put something here.  If you’re using roles, in order to grant/revoke access to a user, simply add or remove the desired role to/from the roles array from the user’s document in the _users database.

Now, CouchDB allows you to create and assign any roles you want, but doesn’t enforce any permissions beyond the aforementioned ‘member’ and ‘admin’ types.  So, since members can both read and write documents, how can you create roles for read-only users?  Unfortunately, CouchDB doesn’t make this easy or obvious, but it does provide a way to make it work.

I’ll illustrate how to do this with an example:  Let’s say you have an application that’s a photo sharing service.  Each user has various ‘albums’ that they can either keep private or share with others.  Most of the time, users are going to want to give others ‘read-only’ access to an album.  So, every time a user creates a new album, we create a database for it.  When a database is created, we also make sure to configure it with three roles: one admin role and two member roles, like so:

{
    "admins": {
        "names": [],
        "roles": [
            "stan_vacation_album:admin"
        ]
    },
    "members": {
        "names": [],
        "roles": [
            "stan_vacation_album:read",
            "stan_vacation_album:write"
        ]
    }
}

So, Stan has a vacation album.  Let’s say he wants to grant read access to his friend Bill.  He tells your app to grant read access to Bill and your app turns around and edit’s Bill’s user document by adding “stan_vacation_album:read” to his list of roles.  Great!  Bill has access, but how do we enforce read-only access and prevent Bill from, say, using Photoshop to edit himself into Stan’s vacation photos (Bill is not a good friend) and saving them over the originals in Stan’s database?

You can create a special design document in each database called ‘_auth’.  Within that design document, you can specify a function called validate_doc_update, which will be triggered each time someone tries to save a new revision of a document.  So, all we need to do is check the user’s role and not allow the update to go through if the user doesn’t have the right ‘role.’  For example:

function(newDoc, oldDoc, userCtx){ 
    if((userCtx.roles.indexOf('stan_vacation_album:write') == -1)&& 
       (userCtx.roles.indexOf('stan_vacation_album:admin') == -1)&& 
       (userCtx.roles.indexOf('_admin') == -1)){
           throw({forbidden: "Not Authorized"});
    }
}

is a quick and dirty function that would do the trick.  Anyone that isn’t an admin or doesn’t have the :write role, won’t be able to write to the database.  Of course you can also do any data validation you like here as well.

Hope this helps 🙂

I’m also developing a CouchDB driver for Go as I work on my project, feel free to use it however you like.

Testing Stripe transactions with Scala

I’m nearing completion on my latest project, Training Sleuth, and I’ve once again decided to use Stripe as my payment processor.  I used Stripe on my previous project, Rhino SchoolTracker, and have absolutely nothing negative to say about it :).

While with Rhino SchoolTracker I tested my stripe payments manually, I decided I wanted to write some automated integration tests this time around (particularly since I’m using Scala for this project which has a decidedly slower and more unwieldy edit-compile-test loop than Ruby does).

The Stripe API is actually split into two parts, a client side javascript library (stripe.js), and a server-side API available in several of the popular server-side web languages (php, java, ruby, python, and javascript (for node.js) last time I checked).  Anyway, the basic concept goes like this:

  1. You serve an HTML payment form (with fields for CC number, etc.) from your server to the client.
  2. When the client submits the form, instead of sending it to your sever, you use stripe.js to grab the form data and sends it to Stripe’s servers, which will validate the card and return a unique token (or an error message in case of invalid/expired credit cards, etc.) via an ajax request.
  3. Once you have the stripe card token, you send it up to your server, do whatever processing you need to do on your end (grant access to your site, record an order, etc.), and then submit a charge to Stripe using the Stripe API.

The key feature of all of this is that the user’s credit card information never touches your server, so you don’t need to worry about PCI compliance and all the headaches that go with it (yes, stripe does require you to use SSL, and despite their best efforts it is possible to mis-configure your server in such a way as to expose user payment info if you don’t know what you’re doing).

Now Stripe offers a test mode, which is what we’ll be using here, with a variety of test card numbers to simulate various conditions (successful charge, declined card, expired card, etc.).  The main problem I ran into writing automated tests in Scala was that I needed to use stripe.js to generate a card token before I could interact with the server-side (Java) API.

Enter Rhino, a Javascript interpreter for Java.  Using Rhino, I was able to whip up some quick-and-dirty javascript to generate a stripe token and call it from Scala.  Of course, Rhino alone wasn’t enough — I also needed to bring in Envjs and create some basic HTML to simulate a browser environment for stripe.js.

First, here’s my stripetest.js:

Packages.org.mozilla.javascript.Context.getCurrentContext().setOptimizationLevel(-1);
load("resources/env.rhino.js");
load("https://js.stripe.com/v2/");
//Stripe REALLY wants some sort of HTML loaded, so here you go:
window.location = "resources/stripetest.html"
Stripe.setPublishableKey('pk_test_PUT-YOUR-STRIPE-TEST-KEY-HERE');
var cardNumber
var token = ""
Stripe.card.createToken({
    number: cardNumber,
    cvc: '123',
    exp_month: '12',
    exp_year: '2016'
},function(status, response){
    this.token = response['id'];
});

And you need to provide some basic HTML, I created a file called ‘stripetest.html’ which merely contained this:

<!DOCTYPE html>
<html>
<head>
    <title></title>
</head>
<body>
</body>
</html>

Simple, but this was enough to get things working.

I dropped these files (along with env.rhino.js which I obtained from the Envjs website) into my test/resources folder.

With all of that in place, I was able to write some specs2 tests:

import org.mozilla.javascript.{Context, ContextFactory}
import org.mozilla.javascript.tools.shell.{Main, Global}
import org.specs2.mutable._

class SubscriptionServiceSpec extends Specification {
//Get Stripe token via Stripe.js
  val cx: Context = ContextFactory.getGlobal.enterContext()
  cx.setOptimizationLevel(-1)
  cx.setLanguageVersion(Context.VERSION_1_5)
  val global: Global = Main.getGlobal
  global.init(cx)

  "'SubscriptionServiceActor'" should{
    "handle Create Subscription Request with bad credit card" in{
      //stripe test pattern to simulate bad card number
      global.put("cardNumber", global, "4000000000000002")
      Main.processSource(cx, "resources/stripetest.js")
      val badToken: String = global.get("token", global).toString
      //Now test and make sure you handle a bad credit card correctly 
    }
    "handle Create Subscription Request with valid credit card" in{
     //stripe test pattern to simulate valid credit card number
     global.put("cardNumber", global, "4242424242424242")
     Main.processSource(cx, "resources/stripetest.js")
     val stripeToken: String = global.get("token", global).toString
     //Now test that you can actually take money from customers 🙂
    }
}

There you go, kind of painful to get set up, but definitely nice to have.