Form plus Logic

Adding jQuery to Sandbar Forms

2010-09-22T23:41:00.000-07:00

The previous post, which introduced sandbar.forms, covered the basics of creating server-side forms. This post will focus on the client and how one may easily integrate these forms with a client-side JavaScript library like jQuery.

The starting point will be a simple form for entering information about programmers. It will have text fields for the programmer's name, hire date and favorite programming language. The form will work as in the previous example; each field will be validated on the server after the form is submitted.

This form will be modified to have a datepicker, an autocomplete field for the programming language and will be validated while it is being filled in. To make it even more interesting, the autocomplete field will be populated with programming languages scraped from GitHub using enlive.

The starting point is shown below and is available in the Sandbar Examples project.


(ns sandbar.examples.fancy-form-before
  "A form which will be made fancy by the addition of jQuery."
  (:use [ring.adapter.jetty :only [run-jetty]]
        [ring.middleware.file :only [wrap-file]]
        [hiccup.core :only [html]]
        [hiccup.page-helpers :only [doctype link-to]]
        [compojure.core :only [defroutes GET]]
        [sandbar.core :only [icon stylesheet javascript]]
        [sandbar.stateful-session :only [wrap-stateful-session
                                         set-flash-value!
                                         get-flash-value]]
        [sandbar.validation :only [add-validation-error
                                   build-validator
                                   non-empty-string]])
  (:require [compojure.route :as route]
            [sandbar.forms :as forms]
            [sandbar.examples.database :as db]))

(defn layout
  ([content] (layout content []))
  ([content javascripts]
     (html
      (doctype :html5)
      [:html
       [:head
        (stylesheet "sandbar.css")
        (stylesheet "sandbar-forms.css")
        (icon "icon.png")]
       [:body
        (if-let [m (get-flash-value :user-message)] [:div {:class "message"} m])
        [:h2 "Sandbar Form Example"]
        content]])))

(defn home []
  (layout
   [:div
    (link-to "/developer/edit" "Add Developer")
    [:table
     [:tr
      [:th "Name"] [:th "Hire Date"] [:th "Favorite Language"] [:th ""]]
     (map #(let [{:keys [id name hire-date language]} %]
             [:tr
              [:td name] [:td hire-date] [:td language]
              [:td (link-to (str "/developer/edit/" id) "Edit")]])
          (db/all-users))]]))

(def properties {:name "Name"
                 :hire-date "Hire Date"
                 :language "Language"})

(forms/defform fancy-form "/developer/edit"
  :fields [(forms/hidden :id)
           (forms/textfield :name)
           (forms/textfield :hire-date {:size 10})
           (forms/textfield :language)]
  :load #(db/find-user %)
  :on-cancel "/"
  :on-success #(do
                 (db/store-user %)
                 (set-flash-value! :user-message "Developer has been saved.")
                 "/")
  :validator #(non-empty-string % :name :hire-date :language properties)
  :properties properties)

(defroutes routes
  (fancy-form (fn [request form]
                (layout form)))
  (GET "/" [] (home))
  (route/not-found "Not Found"))

(def application-routes (-> routes
                            wrap-stateful-session
                            (wrap-file "public")))

(defn run []
  (run-jetty (var application-routes) {:join? false :port 8080}))

The home page will display a table of programmers and will have links to add and edit them. The form is shown below.

If the form is submitted without filling in the data, the error messages shown below will be displayed.

With this, we have a pure server-side form. It will now be enhanced by the addition of JavaScript.

Add jQuery

The first step in improving this form will be to add jQuery. The requirements include jQuery itself, the UI library containing the datepicker and autocomplete, and the CSS for the theme of choice. In this example, the ui-lightness theme is used.

To bring these new resources into the project, the layout function is updated to include the additional style sheet and JavaScripts.


(defn layout
  ([content] (layout content []))
  ([content javascripts]
     (html
      (doctype :html5)
      [:html
       [:head
        (stylesheet "sandbar.css")
        (stylesheet "sandbar-forms.css")
        (stylesheet "ui-lightness/jquery-ui-1.8.4.custom.css")
        (icon "icon.png")]
       [:body
        (if-let [m (get-flash-value :user-message)] [:div {:class "message"} m])
        [:h2 "Sandbar Form Example"]
        content
        (map javascript
             (concat ["jquery-1.4.2.min.js" "jquery-ui-1.8.4.custom.min.js"]
                     javascripts))]])))

Notice that this function has been implemented to allow additional JavaScript files to be passed as arguments. The functions stylesheet and javascript are part of Sandbar and assume that files are located in public/css and public/js respectively.

This form will also require some custom JavaScript which will be put into a file named fancy-form.js. It should only be included on the form page so it will be passed as a parameter to the form layout as shown below.


(fancy-form (fn [request form]
                (layout form ["fancy-form.js"])))

Add a Datepicker

In order to work with the form elements from JavaScript, an id must be set for each of them. This may be done by passing a map of attributes to each field, as shown below.


:fields [(forms/hidden :id)
         (forms/textfield :name {:id :name})
         (forms/textfield :hire-date {:id :hire-date :size 10})
         (forms/textfield :language {:id :language})]

In this step, only the id for :hire-date needs to be added; but it is just as easy to add them all.

Adding a datepicker to a text field is simple with jQuery requiring only a single line of code other than the "document ready" function.

The resulting form with activated datepicker is shown below.

Add Autocomplete

The next improvement will be to make the language field an autocomplete field. The list of languages that will populate the autocomplete suggestions will be scraped from the Languages page on GitHub.

Enlive makes this simple, but first it will be included in the require section of the namespace form.


[net.cgrand.enlive-html :as enlive]

Create a function to scrape the language names, memoize it for performance and then create a function to get all names starting with a specific string.


(defn langs-from-github []
  (let [page (-> "http://github.com/languages/"
                 java.net.URL.
                 enlive/html-resource)
        hits (enlive/select page [:div#languages :li :a])]
    (mapcat #(:content %) hits)))

(def langs (memoize langs-from-github))

(defn langs-starting-with [s]
  (let [pattern (re-pattern (str "^" s ".*"))]
    (filter #(re-find pattern %) (langs))))

Each jQuery UI element has many features and options. This example will be simple, providing only the relative URI which will return a JSON formatted array of languages. jQuery will add a request parameter named "term" to the request which will need to be passed to langs-starting-with in order to filter the list. Since JSON is required, support for JSON will be added to the require section of the namespace form.


[clojure.contrib.json :as json]

A function is required that will get the "term" from the request and then return the JSON formatted array of filtered languages.


(defn langs-autocomplete [{params :params}]
  (let [term (get params "term")]
    (json/json-str (map #(hash-map :value %) (langs-starting-with term)))))

Up to this point, nothing has been done which requires support form sandbar.forms. In fact, no support is required to get this to work. All that is required is a route that will call the langs-autocomplete function and pass it the request. As a convenience, defform does provide some help here in the form of the :ajax option. This option is passed a sequence of function symbols for which it will create routes.


:ajax [langs-autocomplete]

This will actually create two routes: "/developer/edit/langs-autocomplete" and "/developer/langs-autocomplete" which will allow a single line addition to fancy-form.js to enable autocomplete on both the add and edit forms.

With these change in place, the language field will now provide suggestions as users type.

Add Live Validation

In the original code, a validator was added to the form which will ensure that values are entered in each field and display an error message if the values are missing. The validator is shown below.


#(non-empty-string % :name :hire-date :language properties)

It is always good to validate on the server, even if client-side validation is performed with JavaScript. It would not be good if the validation code had to be repeated on both the client and the server; especially when that validation code is complex. defform will allow the client to use the server-side validator via AJAX. To enable this, use the :ajax-validate-at option which is set to a name that will be used to create a route for accessing the validator.


:ajax-validation-at "validate"

This will create two routes following the same pattern as the :ajax option; allowing the client to use the relative path "validate" from both the add and edit form.

The desired behavior is that any time the user moves from one field to the next, the previous field will be validated. This may be implemented as shown below in the final version of fancy-form.js.

What does this do? The validateField function is called to validate a specific field, passing it the field's id. The value for the field is retrieved and sent via AJAX using the relative URI with parameters "validate?y=x" for some id y and value x. This URI was configured above with :ajax-validation-at. The response that comes back from this request will have two keys: status and errors. If the status is "fail" and there is an error for the field being validated, that error will be displayed. It could be displayed in any way, but in this example it is displayed in the previously hidden div for that field. The error div will have an id that starts with the field's id and ends with "-error". More than one field could be validated at a time by simply passing additional parameters in the validate request.

CSS Formatting

At this point there is one annoying behavior which occurs when the name field does not pass validation. The error message causes all the fields to move down after the datepicker has been displayed. For interactive validation it would be better if the error messages did not cause the fields to move. This can be done by adding the following CSS to this project.

With this change, the error messages will be placed next the field label and aligned to the right.

The goal has been reached. The form now provides a datepicker, autocomplete and will be validated as the user enters data.

Conclusion

sandbar.forms does not favor any particular JavaScript library. The only help that it provided here was in allowing easy access through AJAX to the validator function as well as any arbitrary function. I don't plan to take this library too far beyond that or to integrate more tightly with JavaScript. There are too many options for what can be done on the client with JavaScript. When a library starts to help you here, it usually only limits what you can do.

The complete code for this example is located in the Sandbar Examples project.

Taming HTML Forms with Clojure

2010-09-19T16:18:00.000-07:00

HTML Forms may not be interesting but they are tedious. This post attempts to show you how to turn this tedious problem into an insignificant one.

The main thing that I hate about creating forms is that you usually have to do five things before you have a working form. I like it when I only have to do one thing to get something working; I like to see what I am working on and then iterate.

As of version 0.3.0, Sandbar provides support for forms, form layout and form validation. This post will show you how it works.

Imagine that we have the following code as a starting point. All of the namespaces and functions that we will use are listed here so you don't get confused later on.


(ns sandbar.examples.user-form
  (:use [ring.adapter.jetty :only [run-jetty]]
        [ring.middleware.file :only [wrap-file]]
        [compojure.core :only [defroutes GET]]
        [sandbar.core :only [get-param]]
        [sandbar.stateful-session :only [wrap-stateful-session
                                         set-flash-value!]]
        [sandbar.validation :only [build-validator
                                   non-empty-string
                                   add-validation-error]])
  (:require [compojure.route :as route]
            [sandbar.forms :as forms]
            [sandbar.examples.database :as db]
            [sandbar.examples.views :as views]))

(defroutes routes
  (GET "/" [] (views/home))
  (route/not-found "Not Found"))

(def application-routes (-> routes
                            wrap-stateful-session
                            (wrap-file "public")))

(defn run []
  (run-jetty (var application-routes) {:join? false :port 8080}))

The objective of this post is to create a form which may be used to manage users for an application. The main view of said application displays a table of users and has a link to "/user/edit" which should open a form for creating a new user. For each user in the table, a link to "/user/edit/:id" should open a form where one may edit an existing user with this id. For example, "/user/edit/7" would display a form to edit the user with id = 7.

Creating a simple form

Sandbar supports an iterative approach to building forms. For the first iteration a form is defined with a hidden :id field and a textfield for the :username. This definition is created using the defform macro. With each iteration, new options will be added to this definition.


(forms/defform user-form "/user/edit"
  :fields [(forms/hidden :id)
           (forms/textfield "Username" :username)])

defform creates a route-generating function. Below, this function is used to produce the routes that are needed for the user-form.


(defroutes routes
  (user-form (fn [request form] (views/layout form)))
  (GET "/" [] (views/home))
  (route/not-found "Not Found"))

The route-generating function that is created by defform takes a layout function as a parameter. Here we use our layout to layout the form page. After reloading the application, a click on the link to "/user/edit" will reveal the form shown below.

That's not bad for four lines of code. But this form doesn't really do anything. If we click on the buttons, nothing happens. To make it do something useful, defform will need a little more information.


:on-cancel "/"
:on-success #(do
               (db/store-user %)
               (set-flash-value! :user-message "User has been saved.")
               "/")

The current form has two buttons: Submit and Cancel. The :on-cancel option takes a URL where the user will be redirected when this button is pressed. The :on-success option takes a function of the form data which will return the URL to be redirected to when when this button is pressed.

In the function for :on-success, the data is saved and a message is stored in the Flash before returning this URL. This function will only be called after the form data has been validated. Notice that we could catch errors that may occur at a lower level and then decide where to redirect.

That takes care of form submission and cancellation, but what happens when we attempt to edit a user? The form behaves as if we adding a new user. There is one more option to add to make the form fully operational.


:load #(db/find-user %)

The :load option takes a function of the id. Remember that the id is passed in the link to the edit form ("/user/edit/:id") and can be anything which may be used to lookup the user to edit.

This simple form will now allow our users to add and edit information.

Adding more fields

The form works, but it doesn't collect all of the information that is required. It will be used to enter a password, first and last name and email address for each user as well as select the roles that each user is assigned. As a step forward, additional text fields will be added.

The new list of fields is shown below...


[(forms/hidden :id)
 (forms/textfield "Username" :username)
 (forms/password "Password" :password)
 (forms/textfield "First Name" :first-name)
 (forms/textfield "Last Name" :last-name)
 (forms/textfield "Email" :email)]

...resulting in this form.

It is easy to use strings as labels as has been done above. However, it is better to keep all of this information in one place so that it may be easily changed and perhaps internationalized in the future.

To extract all of the labels, create a map named 'labels'...


(def labels
     {:username "Username"
      :password "Password"
      :first-name "First Name"
      :last-name "Last Name"
      :email "Email Address"})

..and add it to the form definition using the :properties option.


:properties labels

The field list may now be written in a much more concise format.


[(forms/hidden :id)
 (forms/textfield :username)
 (forms/password :password)
 (forms/textfield :first-name :last-name :email)]

To select roles, a more complex field in required. A multi-checkbox is a set of checkboxes that allow the user to make multiple selections for a single form field. Creating these can be a lot of work, but with Sandbar it is one line of code. The following line is added to the end of the field list.


(forms/multi-checkbox :roles (db/all-roles) identity)

Here we are creating a form element named :roles using the sequence that we get back from calling (db/all-roles). The final argument is a function that will return the value that will be selected. (db/all-roles) will return a set of keywords #{:user :admin} so the identity function is used to simply return the keyword. If the data returned from (db/all-roles) were more complex, an arbitrary function could be used to generate a value from each element. The result of making the above change, and adding entries to the labels map for :roles, :user and :admin, is shown below.

This form doesn't just display the checkboxes, it actually works. The code required to marshal the data from the request parameters is automatically generated based on the field types and the data that is passed to each field's constructor. Sandbar also provides support for normal checkboxes, textareas, selects and multi-selects. Custom form elements may also be created.

Adding form validation

The form should have two required fields: username and password. Furthermore, the password should have at least 10 characters.


(defn password-strength [m]
  (if (< (count (:password m)) 10)
    (add-validation-error m :password "Passwords must have > 10 characters.")
    m))

(def user-form-validator
     (build-validator
      (non-empty-string :username :password labels)
      :ensure
      password-strength))

Two functions have been created: password-strength will look at the value of the password and make sure that it has more than 10 characters, user-form-validator is a compound validator. The build-validator macro creates validator functions from other validator functions. The :ensure keyword is used to indicate that we would like all of the previous validations to pass before attempting to do the third validation. The third validation is dependent on the other two. To learn more about form validation, see the Form Validation page in the Sandbar Wiki.

To use this validator, add it to the form definition as shown below.


:validator user-form-validator

After making the above changes, the form below is produced.

Notice that there is an asterisk indicating which fields are required. Trying to submit an empty form will generate the following error messages.

Entering a username and then a password with less than 10 characters will generate this error message.

Layout and Style

Having all of the fields in a single column is not ideal. It would be nice if the first and last name fields were side-by-side. To accomplish this, add a layout vector to the form definition.


:field-layout [1 1 2]

The meaning of this vector is that there should be one field in the first row, one in the second and two in the third. Any missing rows will have one element by default so they may be omitted here.

I also like my forms to have a particular style, with a header and footer. This style is built-in and named :over-under. It may be added like this:


:style :over-under

The form is coming along.

It would be better for the submit button title to read "Save" and for there to a "Save and New" button so that new users may be added quickly. It would also be nice to have a better title. To make these changes add the following two options to the form definition.


:title #(case % :add "Add New User" "Edit User")
:buttons [[:save] [:save-and-new "Save and New"] [:cancel]]

The final version of the form is shown below.

And here is the form definition with all of the changes that were made above.


(forms/defform user-form "/user/edit"
  :fields [(forms/hidden :id)
           (forms/textfield :username)
           (forms/password :password)
           (forms/textfield :first-name :last-name :email)
           (forms/multi-checkbox :roles (db/all-roles) identity)]
  :load #(db/find-user %)
  :on-cancel "/"
  :on-success #(do
                 (db/store-user %)
                 (set-flash-value! :user-message "User has been saved.")
                 "/")
  :properties labels
  :validator user-form-validator
  :field-layout [1 1 2]
  :style :over-under
  :title #(case % :add "Add New User" "Edit User")
  :buttons [[:save] [:save-and-new "Save and New"] [:cancel]])

Extending a form

In some circumstances a form may need its behavior to change at runtime. For example, showing an additional field when the current user is an administrator. Sandbar allows us to do this by extending a form. As a quick example, the form above may be extended so that an additional field is displayed and validated only when a user is being edited.


(forms/extend-form user-form :with edit-form
    :when (fn [action request form-data]
            (or (= action :edit)
                (get-param (-> request :params) :notes)))
    :fields [(forms/textarea :notes {:rows 5 :cols 70})]
    :validator #(non-empty-string % :notes labels))

The function under the :when key is a predicate which will determine when these changes should be applied to the form.

Other options

Arbitrary HTML attributes may be set on any of the form elements by passing a map of attributes as shown above when creating the :notes textarea.

There are two other options to defform that were not covered here: :default and :marshal. These two options allow for setting default values for the form and wrapping the marshal function in order to gain fine control of how data is collected from the request parameters.

For other and more detailed information, see the documentation in the Sandbar Wiki and also take a look at the example form code. The complete source code for this example is located in the Sandbar Examples project.

Conclusion

This feature of Sandbar may not be what you need all of the time but when it does work for you, it can be a big help. It can be a huge timesaver when creating prototypes and for building administrator interfaces. There is a lot that can be done to make this even better. For example, it would be nice if there were more form styles available and more options for field layouts. Given some time, it will have this and much more.

I think this shows the potential of Clojure as a great web programming language. It is an example of how macros may be used to encapsulate a pattern that would normally be repeated over and over.

Securing Clojure Web Applications with Sandbar - Part 2

2010-08-23T06:39:00.000-07:00

In part 1 of this series, a simple authorization scheme was added to a small Clojure web application. Continuing with this example, this post will demonstrate how features of sandbar.auth may be used to add form-based authentication and channel security.

Form-based authentication and channel security

If you would like to follow along:


$ git clone git://github.com/brentonashworth/sandbar-examples.git
$ cd sandbar-examples/security
$ open src/sandbar/examples/part_two/start.clj
$ lein deps

This code is a bit different from what we ended with last time. A stylesheet has been added and the layout has been improved. The complete source for our starting point is shown below. Make sure you understand this code before moving on.


(ns sandbar.examples.part-two.start
  (:use (ring.adapter jetty)
        (ring.middleware file)
        (compojure core)
        (hiccup core page-helpers)
        (sandbar core stateful-session auth)))

(defn query [type]
  (ensure-any-role-if (= type :top-secret) #{:admin}
                      (= type :members-only) #{:member}
                      (str (name type) " data")))

(defn layout [content]
  (html
   (doctype :html4)
   [:html
    [:head
     (stylesheet "sandbar.css")
     (icon "icon.png")]
    [:body
     [:h2 "Sandbar Security Example"]
     content
     [:br]
     [:div (if-let [username (current-username)]
             [:div
              (str "You are logged in as " username ". ")
              (link-to "logout" "Logout")])]]]))

(defn data-view [title data & links]
  [:div
   [:h3 title]
   [:p data]
   (if (seq links) links [:div (link-to "home" "Home")])])

(defn home-view []
  (data-view "Home"
             (query :public)
             [:div (link-to "member" "Member Data")]
             [:div (link-to "admin" "Admin Data")]
             [:br]
             [:div (cond (any-role-granted? :admin)
                         "Hello administrator!"
                         (any-role-granted? :member)
                         "Hello member!"
                         :else "Click on one of the links above to log in.")]))

(defn member-view []
  (data-view "Member Page"
             (query :members-only)))

(defn admin-view []
  (data-view "Admin Page"
             (query :top-secret)))

(defn permission-denied-view []
 [:div
  [:h3 "Permission Denied"]
  [:div (link-to "home" "Home")]])

(defroutes my-routes
  (GET "/home*" [] (layout (home-view)))
  (GET "/member*" [] (layout (member-view)))
  (GET "/admin*" [] (layout (admin-view)))
  (GET "/logout*" [] (logout! {}))
  (GET "/permission-denied*" [] (layout (permission-denied-view)))
  (ANY "*" [] (layout (home-view))))

(defn authenticate [request]
  (let [uri (:uri request)]
    (cond (= uri "/member") {:name "joe" :roles #{:member}}
          (= uri "/admin") {:name "sue" :roles #{:admin}})))

(def app (-> my-routes
             (with-security authenticate)
             wrap-stateful-session
             (wrap-file "public")))

(defn run []
  (run-jetty (var app) {:join? false :port 8080}))

Encryption

Our application must ensure that passwords are not sent across the network in plain text. In a production environment, one would enable SSL support on the web server and purchase a legitimate SSL certificate from a valid authority for use with the site's domain. For development, it is good enough to create a self-signed certificate and turn on Jetty's SSL support.

Use Java's keytool to create a self-signed certificate.


$ keytool -genkey -alias sandbar -keyalg RSA -keystore my.keystore -keypass foobar
Enter keystore password:  
Re-enter new password: 
What is your first and last name?
[Unknown]:  localhost
What is the name of your organizational unit?
[Unknown]:  dev
What is the name of your organization?
[Unknown]:  clojure
What is the name of your City or Locality?
[Unknown]:  New York     
What is the name of your State or Province?
[Unknown]:  New York
What is the two-letter country code for this unit?
[Unknown]:  US
Is CN=localhost, OU=dev, O=clojure, L=New York, ST=New York, C=US correct?
[no]:  y

For this example, the password "foobar" was entered. keytool has created a keystore in the file named my.keystore. Make sure this file is located in the root directory of the security module (at the same level as the public directory).

To make use of this keystore, update the run function so that it matches the version shown below.


(defn run []
  (run-jetty (var app) {:join? false :ssl? true :port 8080 :ssl-port 8443
                        :keystore "my.keystore"
                        :key-password "foobar"}))

Setting :join? to false will cause the call to run-jetty to return so that the REPL may still be used. The :ssl-port defaults to 443; here we set it here to 8443. Everything else is straight forward. If you are following along, now would be a great time to start a REPL and test that everything is working as expected.


$ lein repl


user=> (use 'sandbar.examples.part-two.start)
user=> (run)

Navigating to https://localhost:8443/ and http://localhost:8080/ confirms that the application may be used over SSL or standard http and that everything works the same as it did before.

Note: You will get a warning message because the certificate that we are using is not legitimate. This is fine for development; do what you need to do to add an exception for this certificate.

Adding form-based authentication

In the last post, the with-security middleware was added and configured to use our authenticate function. In this section, the authenticate function will be replaced with an authentication function from sandbar.form-authentication and a pre-built login form will be added.

Start by adding the required namespaces sandbar.form-authentication and sandbar.validation.

Delete the authenticate function and replace authenticate with form-authentication in our with-security middleware. form-authentication will redirect a user to a login form when that user is not authenticated.

To implement the login form, add form-authentication-routes to the list of routes.


(form-authentication-routes (fn [_ c] (layout c))
                            (form-authentication-adapter))

The parameters to form-authentication-routes are: a layout function and something that satisfies the protocol FormAuthAdapter. The layout function must take two parameters: the request and the content to layout. The FormAuthAdapter protocol specifies two functions which allow us to adapt this component to our system. The functions are load-user and validate-password. load-user takes a username and password and returns a user map. A user map must at least have the keys :username and :roles. validate-password returns a function that can validate the user map created by load-user.


(defrecord DemoAdapter []
  FormAuthAdapter
  (load-user
   [this username password]
   (let [login {:username username :password password}]
     (cond (= username "member")
           (merge login {:roles #{:member}})
           (= username "admin")
           (merge login {:roles #{:admin}})
           :else login)))
  (validate-password
   [this]
   (fn [m]
     (if (= (:password m) (:username m))
       m
       (add-validation-error m "Username and password do not match!")))))

(defn form-authentication-adapter []
  (DemoAdapter.))

This implementation of load-user will simply look at the username to determine if the user is a member or admin. The validate-password implementation will ensure that the username and password are the same. add-validation-error is a function from sandbar.validation which is being used here to display an error message.

(For more information about validators, see the post Clojure Macros Make Me Happy. I will not go into any more detail here.)

After making these changes, saving them and reloading the namespace,


user=> (use :reload-all 'sandbar.examples.part-two.start)

we can return to our application where we should now have an operational login form. Try submitting the form while it is empty. Try entering a username and password that do not match. The form does the right thing in each situation.

Form Customization

At this point you may want to customize the field names on the form as well as the error messages that are displayed. While we are at it, let's make a custom logout landing page. Create a map with keys that correspond to the fields and errors that we would like to update as well as the key :logout-page that indicates where to go after we logout.


(def properties
     {:username "Username"
      :password "Password"
      :username-validation-error "Enter either admin or member"
      :password-validation-error "Enter a password!"
      :logout-page "/after-logout"})

Update the form-authentication-adapter constructor to merge these properties into our DemoAdapter. Because Clojure's records implement the persistent map interface, the form-authentication module uses the FormAuthAdapter as a map to look up field names and error messages for the login form.


(defn basic-auth-adapter []
  (merge (DemoAdapter.) properties))

After making this change, we will see our own field names and error messages displayed on the login form.

Next, create the logout landing page by replacing the empty map that was passed to logout! with properties and then creating a route


(GET "/after-logout" [] (layout (after-logout-view)))

and view.


(defn after-logout-view []
 [:div
  [:h3 "Logout"]
  [:p "You are no longer logged in!"]
  [:div (link-to "home" "Home")]])

In all the changes that have been made so far, a pattern is emerging; add components in the form of middleware or parametrized routes then adapt it to our project. This same pattern will be used in the next section to add channel security.

Channel security

One thing that you may have noticed is that even though SSL is enabled, there is no way to control when it is used and when not. The goal in this application is to secure passwords sent from the client to the server and, because of the additional delay, to use SSL on as few pages as possible. This may be done by adding the with-secure-channel middleware which takes four parameters: the routes to be wrapped, a security configuration, the port and the SSL port. After adding this middleware, the app var definition now looks like this:


(def app (-> my-routes
            (with-security basic-auth)
            wrap-stateful-session
            (wrap-file "public")
            (with-secure-channel security-config 8080 8443)))

The security configuration security-config is a vector of pairs. Each pair is a regular expression literal followed by a configuration. We use a vector here instead of a map because each entry is checked against the current URI in order with the first match being selected.


(def security-config
     [#"/login.*" :ssl
      #".*.css|.*.png" :any-channel
      #".*" :nossl])

Three keywords are used to represent the three kinds of channel security: :ssl, :nossl and :any-channel. The above configuration causes the login screen to always be accessed through SSL, images and stylesheets to go over any channel and everything else to go through standard http.

A future post will show how this same vector may be used to authorized access based on URI patterns.

If you did not follow along, you may want to take a look at the final version of the source for this exmaple.

Conclusion

Progress has been made, but this app is still not secure. Passwords should not be hard coded into an application like this. The application may be improved by having a way to easily assign different passwords to each user and store them securely. There will be at least two more posts in this series; one them will cover some of the new features of Sandbar which can help make this easy for the most common cases. The other will demonstrate how to authorize access to resources based on URI patterns.

Clojure Protocols and the Expression Problem

2010-08-12T21:25:00.000-07:00

The official release of Clojure 1.2 is just around the corner. One of the most significant changes in this release is the addition of protocols and datatypes. Below, we will explore one way that these features may be used to improve your Clojure programs. Specifically, we will look at how protocols may be used to avoid the Expression Problem.

According to Wikipedia, the term "Expression Problem" was coined by Philip Wadler:

“The Expression Problem is a new name for an old problem. The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety (e.g., no casts).”

We experience this problem when we want to add functionally to library code that is outside of our control. In the Java world, wrappers are commonly used to adapt a class to an interface. The problem with this approach is that the identity of the thing has changed and now our system has to deal with two types for the same thing. In the Ruby world, with its open classes, monkey patching may be used to add functions to a class. Let's face it, anything with the word monkey in it can’t be good. Monkey patching changes the class for everyone and can cause unforeseen problems. Both of these approaches add incidental complexity to our code which is the exact thing that we are trying to avoid by using Clojure.

In this post we will go over some working code that is intended to simulate a real life situation where we may use protocols to avoid the Expression Problem. The code is located in the protocol-examples project in the module named expresssion-problem. If you would like to follow along:


$ git clone git://github.com/brentonashworth/protocol-examples.git
$ cd protocol-examples/expression-problem
$ lein deps && lein javac && lein test com.corp.employee-test
$ open src/clj/com/corp/employee.clj

Simulating an integration problem

Imagine that we are creating a system to work with employees. We have created a library, shown below, which performs various payroll and benefit calculations on employee data. As is the custom of Clojure people, employees are represented as maps so that they gain all of the benefits of being "generically manipulable".


(ns com.corp.employee)

(defn- bonus [years performance]
  (* (+ (* years 500)
        1000)
     (/ performance 10)))

(defn- earned-vacation [years]
  (+ 10 (* years 2)))

(defn- vacation-value [rate years]
  (* rate (* 8 (earned-vacation years))))

(defn payroll [employee hours]
  (* hours
     (:rate employee)))

(defn total-payroll
  [coll hours-map]
  (reduce (fn [total next]
            (let [[{rate :rate} hours] (val next)]
              (+ total (* rate hours))))
          0
          (merge-with conj (group-by :name coll) hours-map)))

(defn employee-bonus [e]
  (apply bonus ((juxt :years :perf) e)))

(defn employee-vacation-value [e]
  (apply vacation-value ((juxt :rate :years) e)))

(defn total-emp-benefits [employee]
  (+ (employee-vacation-value employee)
     (employee-bonus employee)))

(defn total-benefits [coll]
  (reduce + (map total-emp-benefits coll)))

(defn make-employee
  [name years perf rate]
  {:name name :years years :perf perf :rate rate})

The problem

Now imagine that we have a new requirement. The existing payroll system will need to be able to use our new library to calculate benefits. They happen to be using Clojure as well but the payroll system is written in Java and cannot be changed. Their system has a Java class named com.company.Employee which contains the information that we need but does not conform to our interface. Specifically, they will need to be able to call total-emp-benefits and total-benefits passing instances of com.company.Employee.

We may start to solve this problem by creating some tests that demonstrate what we would like our system to do (if you are following along, these tests are already included in com.corp.integration-test).


(ns com.corp.integration-test
  (use (clojure test)
       (com.corp employee))
  (import com.company.Employee))

(def e1 (make-employee "jim" 2 8 20))
(def e2 {:name "sue" :years 15 :perf 6 :rate 60})
(def e3 (Employee. "james" 2 8 30M))
(def employees [e1 e2 e3])

(deftest test-employee
  (is (= (.calculatePayroll e3 40)
         1200M)))

(deftest test-total-emp-benefits
  (is (= (total-emp-benefits e3)
         4960M)))

(deftest test-total-benefits
  (is (= (total-benefits employees)
         33100M)))

In these tests we create two employee maps and one employee using the closed Employee class from the payroll system. Our first test is a sanity check; calling one of the methods that exists in Employee. The other two tests pass instances of Employee to functions in our system. The last two tests will not pass because our current implementation expects plain Clojure maps.

Whatever our solution is, we want to ensure that we do not change the caller's contract. There is a lot of code in production that is using our library and we don't want to force our callers to have to change their code.

What is our contract? We have six reporting functions that are designed to work with maps of employee data. We also have a constructor function that provides a more compact way to creating these maps. Our callers may or may not be using this constructor. The tests that we have in place are testing the caller's contract so if we can get all of the tests working without making any changes to them then we have been successful.

Clojure provides two solutions to this problem: multimethods and protocols. Multimethods dispatch on a function of their arguments, protocol functions dispatch on the type of their first argument. As long as we are happy with the form of dispatch, protocols provide the additional benefits of interface organization and faster dispatch. We will choose to use protocols for our solution.

Adding a protocol

We start by defining the Benefits protocol. We choose the functions from our current system that calculate benefits and take an employee as the first argument.


(defprotocol Benefits
 (employee-bonus [this])
 (employee-vacation-value [this]))

Because these functions are defined in our protocol, we can no longer use them as standalone functions. We must implement them for each datatype that we would like them to support. To start with, we want to ensure that Clojure maps continue to work. We use extend-protocol to implement the functions for maps using our existing implementations. If you are following along, add the following form and remove the functions employee-bonus and employee-vacation-value.


(extend-protocol Benefits
  clojure.lang.IPersistentMap
  (employee-bonus [this]
                  (apply bonus ((juxt :years :perf) this)))
  (employee-vacation-value [this]
                           (apply vacation-value ((juxt :rate :years) this))))

After making these changes our original tests should still work.


$ lein test com.corp.employee-test

Next, we solve the problem that we set out to solve by extending the type com.company.Employee with the Benefits protocol. The final extend-protocol form is shown below.


(extend-protocol Benefits
  
  clojure.lang.IPersistentMap
  (employee-bonus [this] (bonus (:years this)
                                (:perf this)))
  (employee-vacation-value [this] (vacation-value (:rate this)
                                                  (:years this)))

  com.company.Employee
  (employee-bonus [this] (bonus (.getYearsWithCompany this)
                                (.getCurrentPerformanceRating this)))
  (employee-vacation-value [this] (vacation-value (.getHourlyRate this)
                                                  (.getYearsWithCompany this))))

Now we may run all of our tests and... Congratulations! Everything works. From the callers perspective, we have added the functions employee-bonus and employee-vacation-value to the Employee class. Notice that we have not globally changed this class, these functions are only available within the context of the com.corp.employee namespace.

Improving performance

Because we have chosen to use protocols, we can make one final improvement to our library. We can create a record based on our protocol and then update the make-employee function to create instances of this record instead of creating maps. This will increase application performance. First we create the new record.


(defrecord StandardBenefits [name years perf rate]
  Benefits
  (employee-bonus [this] (bonus years perf))
  (employee-vacation-value [this] (vacation-value rate years)))

Next, update the make-employee function to create an instance of StandardBenefits instead of a map.


(defn make-employee [name years perf rate]
 (StandardBenefits. name years perf rate))

Run all the tests to confirm that everything works.

Notice that even the functions payroll and total-payroll continue to work when they are passed instances of StandardBenefits even thought they are not part of our protocol and are expecting plain Clojure maps. They work because defrecord provides a complete implementation of a persistent map which is the main advantage of using it over deftype.

If you were not following along then you may want to have a look at the finished version.

Conclusion

The most important thing about what we have done here is what we didn't do. We made no change to com.company.Employee, we didn't change our reporting functions or even the functions payroll and total-payroll which depend on the persistent map abstraction, and we made no change to the caller's contract.

For more information about Clojure's protocols see the datatypes and protocols pages on the Clojure web site. There is also a very informative presentation, "Clojure 1.2 Protocols", by Stuart Halloway.

Securing Clojure Web Applications with Sandbar - Part 1

2010-08-10T09:01:00.000-07:00

Authorization in the model

This is the first in a series of posts that will show how to use Sandbar to secure web applications. Sandbar is a web application library which adds high level abstractions to Compojure and Ring. The sandbar.auth namespace provides code to make role-based security simple.

Security can be divided into two essential parts: authentication and authorization. In this post we will only explore authorization and are therefore getting only part of the security story. In part two we will add encryption, channel security and form based authentication. In part three we will authorize based on URI patterns. Finally, in the fourth installment, we will use Sandbar to create a user administration tool.

The complete code for this example is located in the sandbar-examples repository under security. If you would like to follow along then clone that repository, get the dependencies, start a REPL and open the sandbar.examples.part-one.start namespace in your favorite editor.


$ git clone git://github.com/brentonashworth/sandbar-examples.git
$ cd sandbar-examples/security
$ open src/sandbar/examples/part_one/start.clj
$ lein deps
$ lein repl


user=> (use ‘sandbar.examples.part-one.start)
user=> (run)

We will start with a small Compojure application and, throughout this series, add features to it. The complete source for our starting point is shown below.


(ns sandbar.examples.part-one.start
 (:use (ring.adapter jetty)
       (compojure core)
       (hiccup core page-helpers)))

(defn query [type]
 (str (name type) " data"))

(defn layout [content]
 (html
  [:html
    [:body
    [:h2 "Sandbar Authorization Example"]
    content]]))

(defn data-view [title data & links]
 [:div
  [:h3 title]
  [:p data]
  (if (seq links) links [:div (link-to "home" "Home")])])

(defn home-view []
 (data-view "Home"
            (query :public)
            [:div (link-to "member" "Member Data")]
            [:div (link-to "admin" "Admin Data")]))

(defn member-view []
 (data-view "Member Page"
            (query :members-only)))

(defn admin-view []
 (data-view "Admin Page"
            (query :top-secret)))

(defroutes my-routes
 (GET "/home*" [] (layout (home-view)))
 (GET "/member*" [] (layout (member-view)))
 (GET "/admin*" [] (layout (admin-view)))
 (ANY "*" [] (layout (home-view))))

(def app my-routes)

(defn run []
 (run-jetty (var app) {:join? false :port 8080}))

(The app var is not required here but having it allows us to load this code once and then make all of the changes below without having to restart our REPL.)

This is a complete application which will show three views: the home page will show public data, the member page will show members-only data and the admin page will show top-secret data. The query function retrieves the data that is displayed.

Add sandbar.auth, middleware and create an authentication function

We would like to have two roles: administrators and members. Only administrators may see top-secret data and only members may see members-only data. We would also like our application to have different behavior based on a user's role.

Before we may add any of Sandbar's functionality, we need to first add Sandbar to our namespace declaration.


(:use (ring.adapter jetty)
       (compojure core)
       (hiccup core page-helpers)
       (sandbar stateful-session auth))

Here we have added the sandbar.stateful-session and sandbar.auth namespaces.

Next, we add some middleware.


(def app
    (-> my-routes
        (with-security authenticate)
        wrap-stateful-session))

Our routes have been wrapped with the with-security and wrap-stateful-session middleware. with-security does all of the hard work. Here it has one argument, an authentication function, which we have not written. The wrap-stateful-session middleware is used by with-security to store user information.

authenticate is where you would do whatever you need to do to figure out who the current user is based on the information in the request. This function is very simple, it takes the request map as an argument and returns a single user map or nil. The user map has two keys: :name, which is a string, and :roles, which is a set of role keywords. For this example, we will use the function below.


(defn authenticate [request]
 (let [uri (:uri request)]
   (cond (= uri "/member") {:name "joe" :roles #{:member}}
         (= uri "/admin") {:name "sue" :roles #{:admin}})))

This function will not secure our application but it does demonstrate how simple it is to add your own authentication scheme. For our example, we simply authenticate the user based on the first URI that is accessed. This function will only be called if the current user is not authenticated.

After making these changes we should still have a working application. Reload the namespace in the REPL and try it out.


user=> (use :reload-all ‘sandbar.examples.part-one.start)

There is a lot of new code running, but the functionality has not changed. At least we know we haven't broken anything.

Protecting data in the model

All of our data is still out in the open. To protect it, we could use the ensure-any-role macro. It takes two arguments: a set of roles and a form that is being protected. The protected form will only be executed if the user is authenticated and has one of the roles in the set.


(defn query [type]
 (ensure-any-role #{:member :admin}
                  (str (name type) " data")))

This isn't really what we are looking for. A simple visit to the home page will result in a java.lang.StackOverflowError. Understanding why will help us understand what is going on behind the scenes. ensure-any-role will throw an exception when the current user is not authenticated. The with-security middleware will catch the exception and then call authenticate to authenticate this user. Our authentication function will not authenticate the user becuase it only authenticates visitors to the admin and member pages. Finally, with-security will redirect the user to the home page and the cycle continues.

This is probably a bug which needs fixing, but it does indicate that we have a problem. There is no way for an unauthenticated user to view the public data on the home page. We could start using conditionals but that will get ugly. Instead, there is a better macro to use in this situation named ensure-any-role-if.


(defn query [type]
 (ensure-any-role-if (= type :top-secret) #{:admin}
                     (= type :members-only) #{:member}
                     (str (name type) " data")))

This macro takes an odd number of arguments. The final argument is the form that you are protecting. The other arguments are pairs of predicates and sets of roles. If the predicate is true then the user must be a member of one of the roles in the corresponding set. If none of the predicates are true then the protected code will be run without authentication.

We can safely try it now. If you first click on the "Member Page" link you will see the members-only data. If you then try to click on the "Admin Page" link, nothing will happen. Our data has been protected.

Logout and permission denied

When ensure-any-role or ensure-any-role-if notice that the current user is authenticated but does not have the correct role, it will redirect the user to "/permission-denied". It would be nice if we could see a better permission denied page. Add the following route and implementation of the permission-denied-view function.


(GET "/permission-denied*" [] (layout (permission-denied-view)))


(defn permission-denied-view []
 [:div
  [:h3 "Permission Denied"]
  [:div (link-to "home" "Home")]])

We may also want to logout so that we can easily try out the different roles.


(GET "/logout*" [] (logout! {}))

The logout route will use a function named logout! from sandbar.auth to clear the current user from the session. The parameter to logout! is a map which may contain the key :logout-page. If this key is not found then the user will be redirected to “/” after logout.

The user will need some kind of link to allow them to logout easily. Add the following div to the end of the body section of the layout.


[:div (if-let [username (current-username)]
        [:div
          (str "You are logged in as " username ". ")
          (link-to "logout" "Logout")])]

The current-username function is another function from sandbar-auth. It will get the current user's username from the session or return nil if one does not exist. After reloading in the REPL, we now see the username at the bottom of the page, we may logout and we have a proper permissions denied page.

Checking the role

As a final flourish for this example we will add the following div to our layout underneath the content and above the logout code that we added above.


[:div (cond (any-role-granted? :admin)
            "Hello administrator!"
            (any-role-granted? :member)
            "Hello member!"
            :else "Click on one of the links above to log in.")]

This shows how to use the any-role-granted? predicate. Here we show a different message based on the role of the current user. any-role-granted? may be passed a single role or a set of roles. It will return true if the user is a member of one of those roles.

The complete source code for the new authorized version is located in the namespace sandbar.examples.part-one.complete.

Conclusion

We haven’t actually secured anything in this example. But hopefully you can see how that would be done by creating a different implementation of authenticate. In the next installment, we will look at one way to do this and add form based authentication to this example project as well as encryption and channel security.

For more information on Sandbar, see the wiki and check out the source code. The source for sandbar.auth is only 245 lines. Sandbar is still quite new and needs a lot of work. I hope this series will draw more attention to it and help to make it better.

Better Clojure Test Results on the Command Line

2010-07-16T08:09:00.000-07:00

After letting folks know about Deview earlier this week, some people thought that a command line tool that improved test results would be a better fit for them. Deview has some features that you can't have in a command line tool, and more are planned for the future. However, I can see the value in having a simple tool that does one thing well. Even though I plan to use and continue to improve Deview, I took a day off and created lein-difftest.

lein-difftest is a completely new project that does not depend on Deview. It depends on many of the same libraries as Deview but is much simpler. It may be used as a Leiningen plugin or from the REPL.

To use it from Leiningen you simply add lein-difftest as a dev-dependency. See lein-difftest for more information. Once you have this new library in place, run lein test and you will see the new and improved results. The example below shows two test runs. One without lein-difftest and the other with.

To use it from the REPL there is a helper function named test-ns with takes care of reloading the target namespace and turning off color.


user> (use 'difftest.core)
user> (test-ns 'some.test.namespace)

Robert Hooke

This project is also a great example of using Robert Hooke to extend Leiningen. Before using Hooke, I had to copy the entire leiningen.test/test function into my plugin code so that I could modify one of its function calls. With Hooke, I simple intercept the call to said function and ensure that it has the correct arguments.

Better Clojure Test Results with Deview

2010-07-13T12:16:00.000-07:00

Over the past couple of months I have been doing a lot of Clojure development. During this time there were two things that have stood out as workflow killers.

The first is caused by a great feature of Clojure. Because Clojure has a solid notion of equality, you may easily test arbitrarily large Clojure data structures for equality. The problem comes when they are not equal and you need to find out what is different. Like many of you, I use clojure.test. When a test fails, the actual and expected forms are printed. Finding the difference in two large forms can be a nightmare. To feel this pain, try to find the difference in the forms below.


actual: (not (= {:remote-addr "0:0:0:0:0:0:0:1%0", :scheme :http, :query-params {}, :session {:sandbar.stateful-session/session {:project {:host "localhost", :name "deview/deview-server", :port 9000, :_id 1, :_v 1, :_type :projects}}}, :form-params {}, :request-method :get, :query-string nil, :route-params {"*" "/deview/deview-server"}, :content-type nil, :cookies {"ring-session" {:value "0bcoa4e4-a852-4976-b2e9-a697a48f1ed6"}}, :server-name "localhost", :params {"*" "/deview/deview-server"}} {:remote-addr "0:0:0:0:0:0:0:1%0", :scheme :http, :query-params {}, :session {:sandbar.stateful-session/session {:project {:host "localhost", :name "deview/deview-server", :port 9000, :_id 1, :_v 1, :_type :projects}}}, :form-params {}, :request-method :get, :query-string nil, :route-params {"*" "/deview/deview-server"}, :content-type nil, :cookies {"ring-session" {:value "0bcca4e4-a852-4976-b2e9-a697a48f1ed6"}}, :uri "/stats/deview/deview-server", :server-name "localhost", :params {"*" "/deview/deview-server"}}))

On several occasions I spent so much time looking for differences that I forgot what I was doing before encountering the problem.

Another related issue is encountered when an exception is thrown during compile time or while running a test. Usually several screens of stacktrace are output; I usually only need to see the first line. It takes up a lot of time to switch focus to the terminal window and then scroll up to find that one line that I care about. If I have encountered exceptions on the last few test runs then I also have to be careful not to scroll too far and view the results from a previous test.

These two tasks take up much more than just time. It is frustrating when you are working to solve one problem and in the process need to stop and solve a different problem. Some call this yak shaving. They also have something in common; for both of them, I am trying to manually find something that the computer should be able to find for me.

I decided to solve this problem before moving forward on any of my current projects.

Building Blocks

Smart people have already been working on these problems.

At the June Bay Area Clojure Meetup, George Jahad presented a small library he had created to solve the first problem. The library is named difform. It has two useful functions difform and sort-form: when difform is passed two forms it will display the diff, sort-form will sort a Clojure form so that it can be diffed. This is good but I want to see that diff in my test results without having to copy and paste. When a test fails, I want to immediately see what the problem is without interrupting my train of thought.

Mark McGranaghan’s clj-stacktrace makes stacktraces much better. Using the stacktrace middleware provided with Ring is a big timesaver when working on Ring web applications. It would be great if I could use this on my test result stacktraces for all my projects.

Deview

Deview is what I came up with. It makes use of the above two libraries as well as some great features of clojure.test which make it easy to capture reporting information while tests are running.

Deview runs yours tests and reports test failures and successes. If an exception is thrown, clj-stacktrace is used to clean up the stacktrace. Deview will then filter it down to show relevant (according to me) trace elements. For example:

When a test fails, the difference between the expected and actual results are displayed. The diff below shows the changes in the two forms at the beginning of this post.

I have been using this for a couple of weeks and have found it to be a huge time saver.

For complete instructions on installing and using deview, see the project README.

Deview needs to be able to run your tests, so it has to run with your project’s classpath which contains all of your project’s dependencies. Current features of deview as well as some that are planned for the future, require deview to have quite a few dependencies of its own. In order to mitigate dependency conflicts, I decided on a client/server architecture where the server is small with few dependencies and the client can have any number of dependencies.

Deview also stores information about your projects. All of this information is stored with the client so as not to corrupt your project directory.

The server is the part that you add to your project. It only depends on Clojure and clj-stacktrace. The server will run in its own process with your project’s classpath. It will run your tests, read your project.clj file and read source files. It will not in any way modify anything in your project.

The client is a Compojure web application. It can be cloned from GitHub and run from the REPL.

Features

The current focus is on testing. Once you are up and running you will be able to see a list of all of your test namespaces. You may choose to run tests for an individual namespace or for all of them.

While tests are running, your view will be updated once per second showing the current running namespace and test.

If exceptions or failures occur, you will see a short stack trace or a diff as shown in a previous section.

If you choose to run all tests and experience failures in one namespace you may easily click on that namespace to re-run its tests.

To re-run the last set of tests you simply refresh the page. This allows you to easily re-run tests with a couple of keystrokes (Command-TAB, Command-R on a mac).

The Future

There are many ways that deview can be improved in the future. Some of them are listed here.

Set up a time interval so that tests are automatically run every now and again
Group tests
Run groups in parallel
Add some useful metrics other than lines of code
Browse and search sources
Add links to source in test error messages
Create test configurations where bindings can be applied, for example: set a flag to not run the tests that hit the database
When tests are running automatically; when a test fails in a namespace, only run the tests for that namespace until the issue is fixed then go back to running all tests.
Find all usages of a function in one or more projects
Interface with a good code coverage tool once one emerges

I have more ideas than I have time. If you find deview useful and would like to help make it better then I would love the help. For more information about deview see the project README.

Relational Mapping for Clojure

2010-06-14T09:19:00.000-07:00

In a recent project, I found myself writing a lot of code to manage associations in a relational database. Much of the code was not specific to that particular project so it was moved into its own project. Carte was born.

Carte is an attempt to bring simple relational mapping to Clojure. It is also an attempt to do relational mapping in a way that complements Clojure.

Carte maps tables and associations in a database to nested immutable data structures in Clojure. Query results are nested data structures which, after being modified, may be saved with a single API call.

Imagine that we have the following database structure:

Using Carte's model macro, we can describe this with the following expression.


(model
  (genre [:id :name])
  (track [:id :name]
         (belongs-to :album))
  (album [:id :title]
         (many-to-many :artist)
         (many-to-one :genre))
  (artist [:id :name]))

For the examples below db is a map which contains the database configuration and the model created above.

Carte provides a fetch function to pull data from the database.


user> (fetch db :album :with :artists)
[{:artists
   [{:name "Patrick Carney", :id 24} {:name "Dan Auerbach", :id 26}],
  :title "Magic Potion",
  :id 16}
 {:artists
   [{:name "Jack White", :id 22}
    {:name "Patrick Keeler", :id 23}
    {:name "Brenden Benson", :id 25}
    {:name "Jack Lawrence", :id 28}],
  :title "Broken Boy Soldiers",
  :id 17}
 {:artists [{:name "Jack White", :id 22} {:name "Meg White", :id 27}],
  :title "Elephant",
  :id 18}
 {:artists [], :title "White Blood Cells", :id 19}
 {:artists
   [{:name "Patrick Carney", :id 24} {:name "Dan Auerbach", :id 26}],
  :title "Thickfreakness",
  :id 20}]

More complex queries may be performed. All of the queries below retrieve data in a single query using multiple left joins.


(fetch db :album :with :artists :tracks :genre)
(fetch db :album :with [:artist {:name "Jack*"}] :tracks)
(fetch db :artist :order-by :name :with [:album :with [:track {:name "Call*"}]])

The query function may be used to create pre-compiled queries and to compose queries. This makes it very easy to create template queries that may be modified and used at a later time.


user> (def & (partial query db))
user> (def album-query (& :album :with :artists :tracks :genre))
user> (def ordered-artist-query (& album-query (& :artist :order-by :name)))
user> (fetch db (& ordered-artist-query (& :genre {:name "Rock"})))
[{:id 118,
  :title "Broken Boy Soldiers",
  :artists
  [{:id 158, :name "Brenden Benson"}
   {:id 161, :name "Jack Lawrence"}
   {:id 155, :name "Jack White"}
   {:id 156, :name "Patrick Keeler"}],
  :tracks
  [{:id 272, :name "Hands"}
   {:id 273, :name "Level"}
   {:id 274, :name "Steady As She Goes"}
   {:id 275, :name "Call It a Day"}
   {:id 276, :name "Together"}],
  :genre {:name "Rock", :id 45}}
 {:id 119,
  :title "Elephant",
  :artists
  [{:id 155, :name "Jack White"} {:id 160, :name "Meg White"}],
  :tracks
  [{:id 269, :name "Seven Nation Army"}
   {:id 270, :name "Black Math"}
   {:id 271, :name "Girl, You Have No Faith In Medicine"}],
  :genre {:name "Rock", :id 45}}]

When a fetch is performed, each map in the data structure has associated metadata to indicate the table that it came from and the original values in the map. This allows Carte to figure out which records need to be updated or deleted. For example, the following sequence of expressions will remove all artists from an album. This will cause records in the album_artist table to be deleted.


(def r (-> (fetch-one db :album {:title "Magic Potion"} :with :artists)
           (assoc :artists [])))
(save-or-update db r)

If we were to delete the album, all associated tracks would also be deleted.

This is just a little taste of what Carte can do. This post is an abridged version of the README for Carte on GitHub. For more detailed information, please see that document. If you have MySQL installed, you should be able to easily follow along with the examples.

In the future, I will add support for Derby, PostgreSQL and SQL Server.

If you do take the time to look closely at Carte, please let me know what you think. I would love some constructive criticism.

Binding Saves the Hour

2010-05-03T15:45:00.000-07:00

Clojure's binding form is one part of Clojure that I just didn't get at first. The description in the API docs is clear, but reading it with my background was like telling a blind man that something is red. Since I had never had a feature like this available in a language, I had no experience with how one might actually use it.

If you look at some of my earlier blog posts about unit testing you will see that at that time I was still in the dark about binding. I was struggling to find a way to mock services and was creating functions with def calls, a big Clojure no-no.

I have since changed my ways and now use binding as it should be used when testing to create environments with alternative implementations of functions.

This post is not about that. It is about how binding saved me some yak shaving one day last week.

I started a new web project last week, and was excited to build something new on top of my sandbar library. The project needed to have a secure area where administrators could edit content on the site. This means that I will need to have a way to edit users, passwords, roles etc. The development module of sandbar has some code that creates a user interface for doing just this. It also has some code for creating the tables that I will need to back it in MySQL. If all goes well, I should be able to save myself many hours of work.

During the previous week, I completed getting a stable version of the library into Clojars. I added sandbar to my dependencies and ran lein deps. The first step was to create the tables that I needed in MySQL. Here is the function that does that:


(defn create-tables
  ([] (create-tables nil))
  ([drop]
     (let [db (get-connection)
           drop-fn (partial database/db-drop-table db)
           insert-fn (partial database/db-insert db)
           create-table-fn (partial database/db-do-commands db)] 
       (do
         (if drop
           (do (drop-fn :user_role)
               (drop-fn :role)
               (drop-fn :app_user)))
         (create-user-table create-table-fn)
         (create-role-table create-table-fn)
         (create-role-user-table create-table-fn)
         (insert-role-records insert-fn)
         (insert-user-records insert-fn)
         (assoc-users-with-roles insert-fn)))))

There is a lot of stuff here but the part that curbed my enthusiasm was (get-connection). I had hard-coded the database configuration. That's why this code is in the development module, I thought. I started to think about all the ways I could get around this problem. Copy the code into my project and make the changes, fix the problem in the library and re-deploy, etc.

But then I remembered binding. get-connection is a function that returns a map with a database configuration. All I need to do is use binding to create a new binding for the var get-connection to a function that returns the correct configuration.

So I fired up the REPL and...


(binding [get-connection 
            (fn [] {:connection
                     {:classname "com.mysql.jdbc.Driver"
                      :subprotocol "mysql"
                      :subname "//localhost/the_db"
                      :user "the_user"
                      :password "123456789"}})]
 (create-tables))

Nice job Clojure!

This would have been easy for me to fix because I am in control of both projects. But what if you wanted to use this function right now in your project? I haven't yet fixed the problem. binding provides the workaround.

binding could definitely be used to win a 'most unreadable code ever' contest. It is a powerful feature and can be abused. Part of the difficulty in learning to use binding properly is knowing when not to use it. But I'm glad it's there. In my opinion, it is one more reason to love (or at least like) Clojure.

Clojure Macros Make Me Happy

2010-04-15T08:52:00.000-07:00

Demonstrating the power of macros with an example.

I have been working a bit on validation recently. The goal is to come up with a sane way of organizing validation code, composing validations and reporting errors. This is another one of those solved problems that I am trying to find a better Clojure solution for. While doing this I had a chance to write a couple of Clojure macros that made me happy. This is the story of one of those macros.

Validator theory

The design is simple. A validator is a function of a map which is being validated. The function returns the map unaltered if it passes validation. If there are errors, error messages will be put in the map under a special key. This allows validators to be easily composed.

Validation helpers

The only difference between a validator and validation helper is that a validation helper may have more arguments than just the map to be validated.

For example, a validation helper which will ensure that a value is a non-empty string would have a signature like this.


(defn non-empty-string [m & k] ...)

I can turn this into a validator by wrapping it in a function of a map.


(defn validate-my-key [m]
  (non-empty-string m :my-key))

Compound validators

Most data are complex and will need to have multiple fields validated or multiple validations performed on the same field. Below is an example of creating a validator for a user login form. As a first step, we create a validator function which will ensure that a user has actually entered a username and password. The non-empty-string function will look up the error message that it needs to display in the properties map.


(defn login-validator [properties]
  (fn [m]
    (non-empty-string m :username :password properties)))

There is one more thing that we would need to check before we allow this user to login. We need to ensure that the password that was entered is actually correct. For this we create a password-validator function which is not shown here. We cannot simply add this validation. There is some logic here. It depends on the previous validation passing. We don't need to bother checking the password if the user did not enter one. This leads of a solution like the one shown below.


(defn login-validator [properties]
 (fn [m]
   (let [v (non-empty-string m :username :password properties)]
     (if (= m v)
       (password-validator m)
       v))))

This works, but it is messy. We store the map returned from the first validation and then compare it with the original map. If they are the same then we continue and check the password. If they are different then we know that the previous validation did not pass and we just return the map. What if we had further validations that depended on all three of these validations passing?


(defn login-validator [properties]
 (fn [m]
   (let [v (non-empty-string m :username :password properties)]
     (if (= m v)
       (let [x (password-validator m)]
         (if (= m x)
           (-> m
               final-validation-1
               final-validation-2)
           x))
       v))))

This is starting to get nasty.

Macros to the rescue

In many other languages we would be stuck with the above "pattern" of combining validators. Or we would need to create some kind of validation DSL with varying degrees of ease. Since Clojure is a Lisp we may easily create a new control structure.

The build-validator macro allows us to write the above complex validator in the following way.


(defn login-validator [properies]
  (build-validator (non-empty-string :username :password properties)
                   :ensure
                   password-validator
                   :ensure
                   final-validation-1
                   final-validation-2))

It has the same semantics as the threading macro (->) in the Clojure. It is a significant reduction in lines of code and a huge improvement in clarity. Simply list each validation helper and insert an :ensure keyword at any point where you must ensure that the previous validations have passed before continuing. The function that is produced will take a map as an argument and return the map with any error messages for failing validations.

The actual macro implementation is not shown here, only its effect. If you would like to see it you may go to validation.clj and check it out.

Conclusion

This example shows just one of many great features of Clojure. As amazing as this is, it is not even close to being the best feature of the language. Spend some time with Clojure, it may seem strange, it may frustrate you at first, but in the end it will make you happy to.

Sandbar 0.2.1 is Released

2010-04-07T13:23:00.000-07:00

The problem

There are a lot of great web frameworks out there: Ruby on Rails, Django, Spring MVC etc. Using these frameworks can make one very productive. They are mature, have huge communities, and offer great value.

Why then am I so cold toward them? With all of these benefits, shouldn't I be happy with all of the work I'm getting done?

Imagine that you are an artist and you want to create a work of art. What if I handed you a canvas that had all of the lines already drawn and you just had to pick the colors and paint within the lines. It would still take some skill to be able to pick the correct colors and apply them in the right proportions. Artists with different levels of skill would still end up with vastly different results. You would finish faster since you don't have to go through all of the trouble of thinking of a subject for your painting. But much of the joy would be removed because so many decisions have already been made for you.

This is how I feel when using a framework. In order to be more productive, I sell out and spend my time learning how someone else thinks web applications should be made instead of thinking about how they should be made.

To further see my point, let's take this to the extreme. What if I created a framework where all you had to do to create an application is start up a program and fill in some blanks? It would ask you a bunch of questions, you would answer them and then it would spit out a working application. This would be great for productivity. I would start looking for something else to work on.

I am not criticizing those who use them or even saying that I'll avoid them unreasonably. I am just explaining how I feel when using them.

The solution

I want libraries, not frameworks. A library does something for you. It doesn't tell you how to work. It says "Use me if you want, replace me if you find something better." When you start a new project you start with an empty directory.

Enter Clojure, Ring and Compojure. Clojure is an amazing language. One of only two that I care about right now, the other being Erlang. Ring and Compojure are LIBRARIES to help you build web applications. When you start a new web application using these libraries you start with an empty directory. If you try this, please take a moment and enjoy looking at your blank canvas.

When I write an application using Ring and Compojure I don't feel like I am being told how to work. I feel like I can be creative again.

What is Sandbar?

So now that I am excited once again about web development using a non-framework, I have started working on a project named Sandbar. It is a set of libraries for web applications built with Ring and/or Compojure.

The first release contains code to make it easier to work with session data for projects that do a lot of that. It also contains some code to help with authentication and authorization. These are humble beginnings and I hope to add many more features.

The project also includes sample code which shows how to use each feature in the library. It also includes a working reference application that uses all of the features.

If you would like to know more about this project, please see the project on GitHub and take a look at the Sandbar Wiki.

If you would like to start using it, you can either fork it from GitHub and build it yourself or get the 0.2.1 release from Clojars.

Final note

I am releasing this early because I want to get feedback from others. My motivation for releasing now came from this blog post by Bradford Cross:

http://measuringmeasures.blogspot.com/2010/01/you-release-late-and-infrequently.html

I especially like this part:

"So what's the right attitude? Humility. It doesn't matter how smart and successful and qualified you are, you simply don't know what you're doing. The good news is that nobody else does either, though some are foolish enough to think that they do (and that's why you can beat them)."

It is still early days for software development. We don't yet know the right way to do things. Let's keep exploring and being creative.

Migrating from Compojure 0.3.2 to 0.4.0 and Ring 0.2.0

2010-04-01T11:58:00.000-07:00

I recently migrated a non-trivial application from Compojure 0.3.2 to 0.4.0 and Ring 0.2.0. These are some of my notes. This is not an authoritative migration guide or even a complete one. The list is in no particular order. Please also note that 0.4.0 has not been released yet. Some of this may change. Keep your eye on the Compojure Google Group and the Compojure Wiki.

Please correct me if any of this information is incorrect. I am trying to help people here, not lead them astray.

There isn't much left of Compojure. Just under 170 lines of code. Everything else has either been moved to Hiccup, Clout and Ring or has been dropped in favor of Ring.

My new list of Leiningen dependencies for Compojure looks like this:


  [compojure "0.4.0-SNAPSHOT"]
  [hiccup "0.2.1"]
  [ring/ring-devel "0.2.0-RC2"]
  [ring/ring-httpcore-adapter "0.2.0-RC2"]
  [ring/ring-jetty-adapter "0.2.0-RC2"]
  [ring/ring-servlet "0.2.0-RC2"]

Check for the latest versions as I think that Ring 0.2.0 has been released.

redirect-to is now just redirect and located in ring.util.response
compojure.core now contains all the route goodness from Compojure 0.3.2. i.e. routes, defroutes, GET, POST etc..
Ring does not create a session key until you actually put something in the session. This will cause problems if you are only using the key but not actually storing things.
Ring uses strings for parameter names instead of keywords.
When defining Compojure routes, you no longer get request, params etc. for free. You need to put a binding form after the route string.
```
(GET “*” (handler request)) -> (GET “*” request (handler request))
(GET “*” (handler params)) -> (GET “*” {params :params} (handler params))
```
The binding form is applied to the request so you can pull out whatever you need.
capitalize is gone. You can use the one in the new Clojure string library clojure.contrib.str-utils2 or clojure.contrib.string depending on which version of Clojure you are using. The latter is in the most recent version of Clojure Contrib.

compojure.control.decorate is gone. Use ring style:


(def app
   (-> my-routes
        with-session
        (wrap-file “public”)
        wrap-file-info)

UPDATE: Compojure now includes a wrap! macro.

Use the Ring jetty adapter to run your app.
```
(run-jetty (var app) {:ssl? true :port 8080 :ssl-port 8443
                      :keystore "my.keystore"
                      :key-password "foobar"})
```
Note the (var app) which will allow interactive development.

I haven’t figured out how to run the app with a context path or how to create a servlet. I am assuming both are possible and not too difficult.
serve-file is gone.

Where you once had a route like this
```
(GET "/public/*" (or (serve-file (params :*)) :next))
```
you can remove it and add wrap-file as shown in the example above.
The HTML stuff from Compojure is now in Hiccup. All of the functions have the same names so you just swap this in and everything works. Depending on what you need, you will have to use hiccup.core, hiccup.page_helpers or hiccup.form_helpers. These correspond to the similarly named files in 0.3.2
If you use (wrap-file “public”), you do not have to include the string “public” in the path i.e. (link-to “/css/my.css”) will work.

If you are still reading and would like more help then you can check out the application here. This blog post refers to the 0.1.0 tag for this repository.

Contribute to Clojure

2009-12-14T15:55:00.000-08:00

Would you like to contribute to Clojure? You can. Donate.

The Clojure REPL, JMX and JConsole

2009-12-14T13:59:00.000-08:00

In my last post I pointed out how you can use jconsole (Java Monitoring & Management Console) to watch the JVM while your Clojure program (or any Java program) is running. I needed this last week to solve a memory problem.

An even better use of jconsole is to fire it up while you are developing. Just add the -Dcom.sun.management.jmxremote option to the Java command that you use to start the REPL and then run jconsole. The REPL is interactive. Having the REPL and jconsole at the same time is interactive^2. Imagine, you write some code, then you go to the REPL to give it a try and as soon as you execute it you can see how much memory you are using, how many threads are running, how much CPU is being used, the number of classes that have been loaded, the type of memory that is being used, and much more. You can even force a garbage collection just in case you don't believe that you are using all of that memory. Just to wet your appitiite, here is an example.

Example

Today I was writing some code that extracted word counts from newsgroup text files. I had the code working but wanted to improve the performance. Below are three of the four windows that you see on the Overview page of jconsole.

In the example below, you can see the memory usage as I ran a function four different times in the REPL.

Here you can see number of threads that are being used for each run. The second two runs used pmap instead of map, therefore, more threads.

Here is the CPU usage for each run. Notice that in the second two runs, more of the CPU is being used and execution takes less time.

Each of these runs lasted a while. Normally I would be sitting there waiting to see the result. Now I have something to do.

Clojure and JMX

2009-12-11T12:47:00.000-08:00

There are some folks out there that are using Clojure and only know what they have to about Java. This post is for you.

Last night I was working on a little fun Clojure project that involved reading very large files. While reading and processing a 700MB file, my application crashed complaining that it had run out of memory. I had given it 500MB of RAM. What else does it need? Have you ever been in this situation where you are having Java memory issues and need to know exactly what’s going on? When this happens, your best friend is JMX and jconsole.

jconsole comes with the JDK so you already have it. To use these two together, you first need to start Clojure with the Java option -Dcom.sun.management.jmxremote. Once the app is running you can run jconsole which can be found in the bin directory of the JDK. This will open a window which will allow you to select the Java process to monitor. Once you select a process, you will see something like this:

This allows you to see the internals of the JVM as it is running. Armed with this information you should be able to figure out what is going on. After that, I'm sure you will go back to being mad at Java.

This is easy but I needed to make it a little easier. I updated my clj script to add the option above to Java when I pass it the -Xjmx option.

You can do a lot more with JMX and Clojure. If you wish to get into it in more detail, check out the jmx library in clojure.contrib by Stuart Halloway.

First-Class Functions and Sequences

2009-12-09T13:48:00.000-08:00

In yesterday's post I wrote about the power of first-class functions and gave an example of how their correct use can lead to more reusable and understandable code.

As I continue to review the book Structure and Interpretation of Computer Programs I came across another great example of the point that I was trying to make. This example comes from section 2.2.3 Sequences as Conventional Interfaces. All code has once again been translated to Clojure.

Prelude

Below are some helper functions that we will use:


(defn empty-list? [x]
  (and (list? x) (empty? x)))

(defn fib [x]
  (nth (fibs) x))

(defn square [x]
  (* x x))

I also use the functions fibs, and flatten below which come from clojure.contrib.lazy-seqs and clojure.contrib.seq-utils respectively.

What We Could Do

Imagine that we have a tree that we have constructed as shown below.


(def a-tree (cons (list 1 2) (list 3 4)))

What if we wanted to create a function that will calculate the sum of the squares of the odd numbers in the tree? We could write something like this.


(defn sum-odd-squares [tree]
  (cond (empty-list? tree) 0
        (not (seq? tree)) (if (odd? tree) (square tree) 0)
        :else (+ (sum-odd-squares (first tree))
                 (sum-odd-squares (rest tree)))))

This works for small trees. It is a Clojure version of the Scheme example in SICP. Don't do this. It won't work for large trees. But let's assume that this is a good working implementation.

Now imagine that later on we need a function that will take an integer n and returns a list of all the even Fibonacci numbers Fib(k), where k is less than or equal to a given integer n:


(defn even-fibs [n]
  (loop [k 0
         result []]
    (if (> k n)
        (reverse result)
        (let [f (fib k)]
          (if (even? f)
              (recur (+ k 1) (cons f result))
              (recur (+ k 1) result))))))

On the surface these look like two very different functions. Looking at the functions, it is not obvious how we could extract any common functionality. Reading this code is difficult.

How could we improve this? This is like the situation where you ask someone for directions and they tell you that if they were going there, they wouldn't start here.

What We Should Do - Thinking in Sequences

What if we had started by solving each of these problems as a series of sequence transformations? If we did this then we would realize that these two problems have a lot in common and that Clojure already has all of the abstractions that we need to write very simple and readable versions of these functions.

Here is what our new version of sum-odd-squares looks like:


(defn sum-odd-squares [tree]
  (reduce + 
          (map square 
               (filter odd? 
                       (flatten tree)))))

And here is how we would write even-fibs:


(defn even-fibs [n]
  (filter even? 
          (map fib 
               (range n))))

Notice all that these two functions now have in common. We create an initial sequence. We transform that sequence using map. We filter the sequence. Also notice how little we are actually doing here. There is very little that can go wrong. It feels like it would be a waist of time to actually write a test for this.

It is nothing less than amazing how many problems can be solved using filter, map and reduce while thinking of your data as a sequence. Clojure encourages you to think in sequences, and if you do, it will pay off in a lot less code.

The Power of First-Class Functions

2009-12-08T14:56:00.000-08:00

Many languages now have support for first-class functions, some fully embrace them. Those that embrace them are the functional languages. That's why their called functional. There is great power when first-class functions are truly first-class.

As with any power that we are given, it is one thing to know it's there, it is another thing to understand it and yet another to use it effectively in your programs without abusing it. In this post I would like to go through a great example of this power and its correct use.

Prelude

The example is taken from the book Structure and Interpretation of Computer Programs (SICP) section 1.3.3 Procedures as General Methods. The example code is written in Clojure instead of Scheme. All of my example code is a Clojure version of the examples in that section of the book so that you will be able to compare implementations. This whole exercise is not meant to be practical but illustrative. I am not trying to write the most efficient and robust code. The main point of this post is to show a style of programming that is enabled by the skilled use of first-class functions which allows the developer to work at the correct level of abstraction and create the maximum amount of reusable code.

Before we dive in, we will need a few helper functions to simplify the code below.


(def tolerance 0.001)

(defn sin [x]
  (Math/sin x))

(defn cos [x]
  (Math/cos x))

(defn square [x]
  (* x x))

(defn cube [x]
  (* x x x))

(defn average [x y]
  (/ (+ x y) 2.0))

(defn abs [x]
  (if (< x 0) (- x) x))

What Are First-Class Functions?

I wish I had studied SICP as a student. It would have saved me a lot of frustration. In that book, first-class elements of a language are described as elements which may be:

   named by variables
   passed as arguments to procedures
   returned as the results of procedures
   included in data structures

Below we will look at the use of the first three features.

A Function to Find the Square Root of a Number

Imagine that we are creating some software which will need to solve various types of math problems. We may decide that we need to function to calculate the square root of a number and therefore write the following code.


(defn sqrt [x]
  (loop [guess 1.0]
    (let [good-enough? #(< (abs (- (square %) x)) tolerance)
          improve #(average % (/ x %))]
      (if (good-enough? guess)
          guess
          (recur (improve guess))))))

That’s not bad. It’s efficient and fairly clear. We may, at this point, decide that we are done and move on the next problem. If we program in this manner, we are going to end up with a lot more code than we actually need. The problem with this code is that there are hidden patterns which can be extracted, named and reused. Once we do this, we will not only have all of that reusable code, but we will also have a cleaner version of sqrt and more readable code in general.

Finding the Fixed Point of a Function

Since we are writing a program that is going to be solving math problems, we may also need to find the fixed point of a function. From section 1.3.3 of SICP we have the definition of a fixed point of a function: “A number x is called a fixed point of a function f if x satisfies the equation f(x) = x. For some functions f we can locate a fixed point by beginning with an initial guess and applying f repeatedly”

We can write a function that does this...


(defn fixed-point [f first-guess]
  (loop [guess first-guess]
    (let [next (f guess)]
      (if (close-enough? guess next)
          next
          (recur next)))))

...and try it out by finding solutions to y = cos y and y = sin y + cos y.


; find solution to y = cos y
(fixed-point cos 1.0)
>> 0.7387603198742113

; find solution to y = sin y + cos y
(fixed-point #(+ (sin %) (cos %)) 1.0)
>> 1.259003859740025

This is a useful technique which we are sure to use throughout our software.

What does this have to do with square roots? Well, if we have a number x then we are looking for a number y such that y * y = x. If we divide both sides of this equation by y we get y = x / y. Given an x, we need to find a solution to y = x / y and we may be able to do this with our new fixed-point function.

If we try this...


(defn sqrt [x]
  (fixed-point #(/ x %) 1.0))

...it will not work. If we apply that function to a value of x over and over again we just oscillate between two values forever.

Average Damping

To fix the above problem we can do this.


(defn sqrt [x]
  (fixed-point #(average % (/ x %)) 1.0))

Instead of using the new value of the function as the new guess, we use the average of the old guess and the new value of the function. This is called average damping. We could again stop here. But average damping is a concept that we may use again in our software. We can extract this into a method...


(defn average-damp [f]
  #(average % (f %)))

...and rewrite our sqrt function as shown below.


(defn sqrt
  "solve y -> x / y to find the square root"
  [x]
  (fixed-point (average-damp #(/ x %)) 1.0))

We could also create a cube-root function using the same pattern but finding a solution to y = x / (y * y).


(defn cube-root
  "solve y -> x / (y * y) to find the cube root"
  [x]
  (fixed-point (average-damp #(/ x (square %))) 1.0))

The average-damp function is a function that returns a function. You pass it a function and it returns an average damped version of that function. We now have another powerful concept that we can use anywhere thanks to first-class functions. Also, our implementation of sqrt is much better. As stated in SICP “Notice how this formulation makes explicit the three ideas in the method: fixed-point search, average-damping, and the function y -> x / y.”

Newton’s Method

If we are going to be finding solutions to equations then we will surely be using Newton’s method at some point. From SICP we have this description of Newton’s method.

“If x -> g(x) is a differentiable function, then a solution of the equation g(x) = 0 is a fixed point of the function x -> f(x) where

and Dg(x) is the derivative of g evaluated at x.”

Notice that Newton’s method will use the fixed-point function that we have created. Nice!

To implement Newton’s method we will need a derivative function. Notice that the derivative function will need to be a function that takes a function g and returns a function of x.

Here is my Clojure implementation.


(defn deriv [g]
  (let [dx 0.00001]
    #(/ (- (g (+ % dx)) (g %)) dx)))

We now use this to create the newton-transform function which is also a high-order function...


(defn newton-transform [g] 
  #(- % (/ (g %) ((deriv g) %))))

...and finally we create the newtons-method function using fixed-point.


(defn newtons-method [g guess]
  (fixed-point (newton-transform g) guess))

Newton’s method can find the solution to the equation g(x) = 0. If we wanted to use Newton’s method to find square roots then we would need to find the y such that (y * y) - x = 0.


(defn sqrt 
  "find the fixed point of y -> (y * y) - x"
  [x]
  (newtons-method #(- (square %) x) 1.0))

The Final Abstraction

Notice that our two different implementations of sqrt follow a pattern: find the fixed point of a transformed function based on an initial guess. Let’s extract that pattern for reuse.


(defn fixed-point-of-transform [g transform guess]
  (fixed-point (transform g) guess))

We can now use this to implement sqrt using two different functions and function transformations.


(defn sqrt [x]
  (fixed-point-of-transform #(/ x %)
                            average-damp
                            1.0))

(defn sqrt [x]
  (fixed-point-of-transform #(- (square %) x)
                            newton-transform
                            1.0))

These are clear implementations of sqrt using high level concepts that are reusable in many areas of our software.

Conclusion

Think about the simplicity, conciseness and maintainability of the software that you could write if you could master this kind of programming.

I have already noticed a big difference in the kind of code that I write as I have started to use Clojure. Many of the most commonly used concepts have already been extracted to functions and are included in the Clojure API. Someone who has internalized this API and knows how to use it correctly can do amazing things with very little code. For exmaple, take a look at this code from a blog post by Tim Bray entitled Idiomatic Clojure.


(defn find-widely
  "Return a map of pages to hit counts in filename."
  [filename] 
  (apply merge-with +
         (pmap count-lines
               (partition-all *batch-size*
                              (line-seq (reader filename))))))

In another post entitled Eleven Theses on Clojure Tim, who is otherwise inclined to like Clojure, writes that it is a handicap that Clojure is a Lisp and that some human minds may react poorly to the find-widely function above. This is true for a reader who is new to Clojure and not already familiar with the concepts that are being used. But for someone who has internalized the API, this is more readable than most other ways you could code this same functionality. This is made possible by embracing and mastering the use of first-class functions.

My Clojure Setup - Update

2009-12-01T11:06:00.001-08:00

Update: There is now an official Getting Started page for Clojure. Please refer to the instructions on this page for information about how to get started with Clojure in the environment that you are most comfortable with. The information below may now be irrelevant.

I recently posted about my Clojure setup because I thought that it would be useful to others who are getting started with Clojure. I have made a couple of changes to the way I am doing things and thought that it might be time for an update.

The Ruby Script

There have been two changes to the Ruby script that I use to start Clojure. The previous version did not check for command line arguments correctly. This has been fixed. Next, I made a small improvement. I don't like having to type ".clj" each time I run a script so I added some code to add the file extension if you leave it out.

I have included the entire script below.

This isn't meant to be used "as is" on your system. If you plan to use it then you will need to make the necessary modifications to get it working. This is what works for me so I thought I would share.

Running Tests

I have also made improvements to the code that I use to run my tests. See my post on Unit Testing for more information.

I like for my tests to run fast. Clojure makes concurrency easy. Therefore, I have changed my test runner to run each test namespace in parallel. Sometimes I only want to run tests for a single namespace. I can now do this by adding the file name, without the .clj extension as an argument to the script which runs my tests. Finally, if I add "-seq" as a command line argument, all of the test namespaces will be run sequentially.

Here is the code:

This file is part of a small (but growing) set of utilities that I use which can be found on GitHub.

If you are going to run your tests in parallel then they have to be able to run in parallel. It is good practice to set your tests up so that they don't depend on each other. Actually running them in parallel forces you to do this.

I only have two cores on my development machine and this is already saving me time. The more cores that I have in the future the more time it will save. It feels good to know that newer hardware will once again cause my programs to run faster.

Programming Plato’s New Computer

2009-11-24T15:17:00.001-08:00

There are four things that have a huge impact on how we think about solving problems as software developers: our experience, education, programming languages and Plato’s computer.

Plato’s computer is the ideal computer. Not the best computer, but the idea that we have in our minds about what a computer is and how it works. Our programming languages are not yet so high level that we don’t have to think about Plato’s computer when solving a problem. After all, we’re programmers, and we have to program something. We are programming Plato’s computer.

So what does Plato’s computer look like these days? It has a processor, RAM, a disk, a network connection, and I/O capabilities.

Computer technology changes very rapidly. But notice that Plato has been using the same computer for a very long time. I got a Commodore 64 back in the 80s and learned to program it using BASIC. Real computers have changed a lot since then. But Plato still has the same one.

Sometime in the next 20 years Plato will be getting a new computer. There will only be one additional letter in the description of his new machine. An “s”. It will have processors, RAM, a disk, a network connection, and I/O capabilities.

That’s right, it will now have processors.

In this business, if something has been the same for 30 or 40 years, and then changes, that is a big deal. If one of the four things that impacts the way that we think about solving problems changes, that is a big deal. This is what we call a paradigm shift.

Once Plato gets his new computer, we will all be doing things differently. After all, that is the cause of him getting a new computer.

Most of us now have computers with multiple processors and/or cores. My laptop has two. My Windows desktop machine has four. My Linux machine has eight. Maybe you know some really cool guy who has thirty two. The other day I read about a machine that has 255,584 (Unnecessary nerd fact: this machine also has 362 terabytes of RAM and a 10 petabyte file system.) Even though these machines exist, it has not penetrated through to the ideal. It won’t until most machines have thousands of cores. Today most machines have one or two cores. Some day most will have thousands. Some day millions.

A huge change is coming. It has already started and will be complete sometime in the middle of my career. As with all paradigm shifts, there will be denial, crisis and maybe even despair.

As for me, I would like to start wrapping my head around this as soon as possible. Many are already doing this. I want to learn how to program Plato’s new computer. At the very least it will mean shifting my thinking from sequential to parallel problem solving, investigating new (to me) languages such as Clojure, Erlang and Haskell and questioning common architectural patterns which are based on the limitations of Plato's old computer.

What do you think are the most significant changes that will occur as we move to this new paradigm?

My Clojure Setup

2009-09-12T16:04:00.000-07:00


clj => starts a REPL
clj some_file.clj => Runs the script some_file.clj
clj some_file.clj arg1 arg2 => Runs the script some_file.clj and passes 
arg1 arg2 to the script

When I start to get serious, I create a "project" which looks like this:


my-project
.cljrc.clj
bin
lib
src
test

The file .cljrc.clj establishes this as a Clojure project. If I execute clj anywhere within this project it will determine the root of the project by locating the .cljrc.clj file and then start the REPL in that directory. It will also load any jar files within the lib directory into the classpath. The file .cljrc.clj is passed as a parameter to the REPL which will execute the Clojure code contained therein.

bin contains commands that allow me to do things like run my tests, compile java code, package java code in a jar file and put it in the lib directory for use from Clojure, etc...

All source files go into src and all tests go into test.

I can run SLIME from emacs and the clj script will be used to start Clojure. Once I have started a REPL using this command. I can open emacs and connect to the running REPL and interact with it from within emacs.

Configure Emacs

I did not use M-x clojure-install to configure emacs. Instead, I followed this tutorial http://riddell.us/tutorial/slime_swank/slime_swank.html which worked for me on OS X even though it targets Ubuntu. After doing this, I had SLIME working with Clojure in emacs.

Here is my .emacs file:


;; clojure-mode
(add-to-list 'load-path "/opt/clojure/clojure-mode")
(require 'clojure-mode)

;; swank-clojure
(add-to-list 'load-path "/opt/clojure/swank-clojure")
(setq swank-clojure-binary "clj")
(require 'swank-clojure-autoload)

;; slime
(eval-after-load "slime"
`(progn (slime-setup '(slime-repl))))

(add-to-list 'load-path "/opt/slime")
(require 'slime)
(slime-setup)

Notice that I added the swank-clojure-binary line instead of swank-clojure-config.

The Ruby Script

This is the Ruby script that I use to start Clojure. I don't do much Ruby coding so please don't laugh.


#!/usr/bin/env ruby -wKU

JAVA_HOME="/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home"
JAVA="#{JAVA_HOME}/bin/java"
GIT_ROOT=ENV['GIT_ROOT']
CLJ="/opt/clojure"
LIB="#{GIT_ROOT}/library"
classpath=".:src:test:classes" +
":#{CLJ}/clojure/clojure.jar" +
":#{CLJ}/clojure-contrib/clojure-contrib.jar" +
":#{CLJ}/swank-clojure" +
":#{LIB}/tools/ant.jar" +
":#{LIB}/tools/ant-launcher.jar" +
":#{GIT_ROOT}/research/clojure/fpl-clojure-util/fpl-clojure-util.jar"
JLINE="#{LIB}/bin-tools/jline-0.9.94.jar"

def find_root(d, repl_config)
Dir.chdir(d)
if Dir.pwd == "/"
nil
else
if Dir.glob("#{repl_config}").size == 1
Dir.pwd
else
find_root("../", repl_config)
end
end
end

repl_config=".cljrc.clj"
dir = Dir.pwd
if find_root(dir, repl_config)
puts "Running in #{Dir.pwd}"
else
Dir.chdir(dir)
puts "Running in #{Dir.pwd}"
puts "Using default #{repl_config}"
repl_config="~/.cljrc.clj"
end

if File.directory? "lib"
Dir.foreach("lib") do |x|
if x =~ /.*.jar$/
classpath = classpath + ":lib/#{x}"
end
end
end

if ARGV[0]
puts "Running Clojure Script... #{ARGV[0]}"
if ARGV[0].size > 1
system("#{JAVA} -Xms500M -Xmx500M -cp #{classpath} "+
"clojure.lang.Script #{ARGV.shift} -- #{ARGV.join(" ")}")
else
system("#{JAVA} -Xms500M -Xmx500M -cp #{classpath} "+
"clojure.lang.Script #{ARGV[0]}")
end
else
puts "Starting Clojure REPL..."
system("#{JAVA} -cp #{classpath}:#{JLINE} jline.ConsoleRunner "+
"clojure.lang.Repl #{repl_config}")
end

Note: This file has been updated in a more recent blog post.

I name this file clj and put it somewhere in my path. If you want to use this then you will need to modify the default libraries that are loaded.

Here is what I have in .cljrc.clj:


(use '[clojure.contrib.duck-streams :only (spit)])
(use '[swank.swank :only (start-server ignore-protocol-version)])

(defn exit [] (. System exit 0))

(defn swank-server []
(ignore-protocol-version "2009-09-08")
(spit (java.io.File. "port.txt") "")
(start-server "port.txt" :port 4005 :dont-close true))

This code will be executed when you run clj within a project. It defines an exit function that you can use to exit the REPL. It also defines a function that will start a swank server from within the REPL. You will need to the set the correct value for ignore-protocol-version. If you don't know what to put here then you can delete this line and emacs will give you a hint as to what it should be.

So how does this all work?

On Mac OS X I have associated the .clj file type with Carbon Emacs. This allows me to open a file for editing by typing open some_file.clj. I can also use wildcards to open multiple files from the command line in emacs.

Once emacs is open, start SLIME by typing M-x slime. This will start the Clojure REPL using the clj script with the classpath just how you want it.

You may also connect to an existing REPL. Start the REPL from the command line using clj. Once the REPL is up type (swank-server). In emacs, type M-x slime-connect. Emacs will ask about the host and port, press enter twice.

That's how I do it for now. I am still working on it. Once I get more comfortable using SLIME I will update this post with any improvements I have made. The one improvement that I do want to make is to change my method of testing to fit with clojure-test-mode so that I can utilize its featues.

Clojure Unit Testing - Part 2

2009-09-07T10:11:00.001-07:00

This is the second part of an entry on unit testing in Clojure. In the first part I covered my test setup and how to test private functions. In this part I will show how I set up the database stuff so that I could easily provide alternative database implementations for testing.

Mocking the database

I have a lot of functions that I would like to test that use the database. Here is one example:


(defn- record-exists?
 "Determine if a record exists in the database that is the same as the passed 
  spending map. A record is the same if it has the same date, amount and ends 
  with the same description.  Records with a description like 'Check 1234' 
  can be changed to 'Car Payment - Check 1234' and will still be recognized as 
  the same record."
 [s]
 (let [coll (db/find-in :spending {:date (:date s) :amount (:amount s)})]
   (if (seq coll)
       (if (seq (filter #(. (:description %) endsWith (:description s)) coll)) 
           true)
       false)))

Note the use of db/find-in to retrieve data from the database. For testing I will need this function to return predictable results. This function is in the namespace com.formpluslogic.budget.main which is setup like this:


(ns com.formpluslogic.budget.main
 (:require (com.formpluslogic.budget
             [db_interface :as db :only (save-or-update 
                                         find-in 
                                         create-budget 
                                         create-spending)])))

com.formpluslogic.budget.db_interface is an interface to the physical database and so relies on a concrete database implementation to do its work. One of the functions in the interface is named find-in:


(defn find-in
 "Find all items in a table or find items using a query"
 ([table] (find-in table {}))
 ([table criteria]
   (add-budget-to-coll (map #(assoc % :type table) (*db-find* table criteria)))))

The physical database implementation is provided by the function *db-find*. There are four of these functions used by db_interface: *db-find*, *db-update*, *db-insert* and *db-delete*. I have an implementation of these functions that interacts with a Derby database but I do not want to hard code this into the interface. Instead I do the following at the top of the db_interface:


(defn use-database [ns]
 (require ns)
 (def *db-find* (ns-resolve (the-ns ns) 'db-find))
 (def *db-update* (ns-resolve (the-ns ns) 'db-update))
 (def *db-insert* (ns-resolve (the-ns ns) 'db-insert))
 (def *db-delete* (ns-resolve (the-ns ns) 'db-delete)))

When the application starts, the first thing that it needs to do is call use-database passing the namespace that contains the implementations of the functions. The functions from that namespace are then mapped to vars that are used in the interface. This allows me to easily use one implementation at run time and another during testing.

Testing printed output

Many of the functions that I want to test generate printed output. I want to be able to check the resulting output and I also don't want the output to be displayed along with my test results. Clojure provides with-out-str for this purpose.


(defn budget-list [] (db/find-in :budget))
(defn pb
 "Print the current budget."
 ([] (pb :item))
 ([field] (ui/display (sort-by #(field %) (budget-list)))))

; Test Printing Budget
(def test-pb-result
 (str
   "  1                 Auto         200.0\n"
   "  2            Bank Fees          25.0\n"
   "  3            Groceries         500.0\n"
   "                                 725.0\n"
   "Total Count: 3\n"))

(deftest test-pb
(is (= (with-out-str (main/pb)) test-pb-result)))

This concludes my description of how I have got around some tricky issues in setting up and performing unit tests in Clojure. I'm still new to Clojure so there may be a better way to do these things. Please let me know what you think.

Clojure Unit Testing - Part 1

2009-08-04T08:34:00.001-07:00

This week I have been writing some Clojure code and starting to think about how to test it. What follows is a description of some of the tricky issues that I am running into while trying to implement a few of these tests and how I solved them. Please use the comments to give me feedback. This blog entry has got quite long so I will be splitting it into a couple of parts.

The code I am testing implements a simple budget system for my wife and I. It's fairly small but it will allow me to learn how to do several interesting things with Clojure.

My test setup

I like to keep my tests separate from the code it is testing. In the root directory of my project I have a src directory and test directory. In each directory I have namespaces under com.formpluslogic.budget. For the namespace com.formpluslogic.budget.main the tests will be in com.formpluslogic.budget.test_main.

Now that I am organized I'll need an easy way to run all of my tests. I would like to be able to run bin/test from the root of my project directory and have all of the tests run. To accomplish this I have created two utilities.

file_utils contains some functions that help in running tests but may also be useful in other situations.


(ns com.formpluslogic.util.file_utils
  (:use [clojure.contrib.str-utils :only (re-split re-gsub)])
  (:import (java.io File)))

(defn remove-file-ext [file-name]
  (let [index (.lastIndexOf file-name ".")]
    (apply str (first (split-at index file-name)))))

(defn file-to-ns-string [f root-dir]
  (let [f-sep File/separator
        td-p (re-pattern (str f-sep root-dir f-sep))]
    (re-gsub (re-pattern f-sep) "."
      (remove-file-ext
        (last (re-split td-p (.getAbsolutePath f)))))))

(defn file-seq-map-filter [dir mf ff]
  (filter ff (map mf (file-seq (File. dir)))))

run_tests will run all of the tests that are found under the test directory.


(ns com.formpluslogic.util.run_tests
  (:use [clojure.test :only (run-tests)]
        [com.formpluslogic.util.file_utils]))

(def test-dir "test")
(def test-file-prefix "test_")
(def test-file-ext ".clj")

(defn- is-test-file? [file-info]
  (let [file-name (first file-info)
        re-s (str test-file-prefix ".*" test-file-ext)
        re-p (re-pattern re-s)]
    (re-matches re-p file-name)))
            
(defn- get-file-info [f]
  [(.getName f) (file-to-ns-string f test-dir)])
                    
(def test-namespaces
  (map #(symbol (last %))
    (file-seq-map-filter test-dir 
                         get-file-info 
                         is-test-file?)))
    
(defn run
  "Runs all defined tests"
  []
  (println "Loading tests...")
  (apply require :reload-all test-namespaces)
  (time (apply run-tests test-namespaces)))

(run)

I started with code that I found in clojure.contrib and made a couple of modifications. The run function should be called from the root directory of the project. It will search the test directory for test namespaces and run them. All I need to do to add new tests is create new test files that start with "test_".

Finally I created a script in the bin directory named test that will run the run_tests file.

Now that I have an easy, low maintenance way to run my tests, I can start writing them.

Testing private functions

The first function that I chose to test was private. So how do you test private functions? After some research I found some possible solutions on the Clojure group. I went with the following approach.

Here is the private function which is located in the com.formpluslogic.budget.derby namespace.


(defn- split-criteria
  "Create a sequence where the first element is the vector
   of keys and the remaining elements are the values"
  [criteria]
  (cons (vec (keys criteria)) (vals criteria)))

My tests are in com.formpluslogic.budget.test_derby which is why I cannot directly access the private function. I use ns-resolve to map the function to a var in the com.formpluslogic.budget.test_derby namespace.


(def split-criteria 
  (ns-resolve 'com.formpluslogic.budget.derby 
              'split-criteria)) 
(deftest test-split-criteria
  (is (= (split-criteria {:id 1}) (list [:id] 1)))
  (is (= (split-criteria {:id 1 :name "John"})
         (list [:id :name] 1 "John"))))

What do you think about this? Is there are better way?

Mocking the database and testing printed output

This application uses a database to store information on disk. When testing, I want to be able to run my tests without using the rdms that I am using for functional and integration testing. I also want to make it easy to use a different database implementation. At the moment I am using Apache Derby. In my next post I will show how I set this up as well as how to test printed output.

Clojure Concurrency - Part 1

2009-07-24T14:54:00.000-07:00

As I continue to slowly go through my copy of "On Lisp" I have come across another interesting piece of example code on page 18 in the section on closures. It illustrates some interesting ideas from functional programming in general and also some very cool Clojure specific ways of dealing with mutable state and concurrency. The Common Lisp version of the code is shown below.


(let ((counter 0))
  (defun new-id ()   (incf counter))
  (defun reset-id () (setq counter 0)))

Two "functions" are produced that share a counter variable. Each call to new-id will increment the counter. A call to reset-id will reset the counter to 0. Notice that I quoted the word "functions". These are not pure functions because they have side effects (mutable state) and do not always produce the same result when called with the same arguments. But that is what makes them interesting. This code produces something that is kind of like an object. It has private state that can only be modified using the interface that it provides. I am eager to see other ways that this construct is used as I continue through the book.

Clojure does not allow you to write code like that shown above. All data is immutable in Clojure so you cannot create a variable and then change its value. That's not totally true. One version of this code ported to Clojure is shown below. This implementation was taken from Fogus' blog.


(let [counter (to-array [0])]
  (defn new-id [] (aset counter 0 (inc (aget counter 0))))
  (defn reset-id [] (aset counter 0 0)))

This code uses Clojure's Java roots to create the mutable state. The (to-array [0]) call creates a Java array. This allows you to change the value in the array without changing the counter itself. This is a trick to accomplish the same thing as the code above. Because Clojure allows integration with Java, you will always be able to do this kind of thing. This codes works. Or does it? Let's write some code to test it.


(defn make-100-ids [id-fun]
  (doall (take 100 (repeatedly id-fun))))

(defn test-c [nthreads id-fun]
  (reset-id)
  (apply max (apply concat
    (apply pcalls (repeat nthreads #(make-100-ids id-fun))))))

(test-c 100 new-id)

The function make-100-ids will call the id-fun 100 times creating 100 new ids. The test-c function will create any number of threads and each thread will concurrently create 100 ids. It will return the highest id that is created. In the last bit of code we call test-c and ask it to create 100 threads using the new-id function. What should we expect to happen? If 100 threads create 100 ids then highest id should be 10000.

If you run this test using the Clojure implementation above you may not get 10000. When I ran this code I got 9785. This means that somewhere along the way more than one thread is getting the same id returned from new-id. The same thing would happen if we tested the Common Lisp version above. These functions are not thread-safe. Clojure gives us a set of tools to make writing thread-safe code easy.

Below I have created a new version of this code that uses refs to manage the mutable state.


(let [counter (ref 0)]
  (defn new-id [] (dosync (ref-set counter (inc @counter))))
  (defn reset-id [] (dosync (ref-set counter 0))))

This still looks more complex than the Common Lisp version and also looks like it has about the same amount of code as the previous Clojure version. So why is this better? It is more complex but that is because it is doing a lot more. Instead of just setting the counter to 0 we set it to a ref which has a value of 0. refs can only be changed within a transaction so we have to surround the code that changes the ref with dosync. This is enforced by Clojure. If you try to change the ref outside of a transaction you will receive an error message. Within dosync the value of the ref will not be altered by any other thread. Within dosync you will always see a consistent view of all your refs. If you think of dosync like a database transaction you will not be far off. The documentation on the Clojure site regarding refs is very good and will explain in detail what refs do for you. I will not repeat that here.

If we run this same test using this new Clojure version we get the correct answer every time.

Update

As Erik points out in the first comment, a better way to write this code would have been to use an atom instead of using ref and dosync. The code is smaller and it runs faster. See comment below.


(let [counter (atom 0)]
  (defn new-id [] (swap! counter inc))
  (defn reset-id [] (reset! counter 0)))

Clojure lazy-seq and Recursion

2009-07-16T17:00:00.000-07:00

Finishing Stuart Halloway's excellent book, "Programming Clojure", has renewed my interest in functional programming. It has also made me realize how much I am wired to think in terms of imperative solutions to problems. I need to delve a little deeper into the mysterious world of functional programming. Therefore, my next reading project will be to go through "On Lisp" while following Fogus' and Stuart's blogs where they port example code to Clojure.

This is day one and my effort has already been fruitful. Today I had to confront the lazy-seq macro.

On page 15 in my version of "On Lisp" is the our-remove-if function which is equivalent to remove in Clojure. Below is the Fogus port to Clojure with one small modification (using lazy-seq instead of lazy-cons which has been removed from Clojure).


1 (defn remove-if [f lst]
2  (if (seq lst) ; idiomatic
3   (if (f (first lst))
4      (recur f (rest lst))
5      (lazy-seq (cons (first lst) (remove-if f (rest lst)))))
6  nil))

This code will work with any size data set. Let's go through this line by line and pay special attention when we get to line 5.

In line 2 we use (seq lst) to determine if the list is empty. If it is then we return nil on line 6 otherwise we evaluate line 3. (seq lst) will return nil if the list is empty.

On line 3 we determine if the first element in the list is something that we want to remove. If it is then we remove it on line 4 by calling (recur f (rest lst)). This is a recursive call of the remove-if function. Why not just use (remove-if f (rest lst))? Clojure does not automatically do tail-call optimization. You have to be explicit and use recur which will allow you to recursively call the function without consuming stack frames with each call. For this to work (and compile) the call to recur has to be in tail position meaning that no executable code appears after the call to recur.

If we are not going to remove the first element then we need to do something like this


(cons (first lst) (remove-if f (rest lst)))

This is basic recursion. We want to create a new list composed of the first item and the result of processing the rest of the list with remove-if. Unfortunately this code will not work with large lists. We will eventually use all of the available stack frames. We also cannot use recur here because it would not be in tail position. cons creates a list from the first element and the result of the recursive call. It's stack frame needs to stick around until that recursive call completes. What to do. What to do...

We need to make one simple change to make this work. Wrap the above code in a lazy-seq.


(lazy-seq (cons (first lst) (remove-if f (rest lst))))

It does work. But why and how does it work? This was hard for me to understand only because I come from a non-functional (or maybe it's dysfunctional) background.

When we make a call to remove-if we don't get a list, we get a function that can create the list, but only if and when we need it. lazy-seq creates a logical sequence but does not execute any of the code in its body until it has to. Imagine calling remove-if with a list where all of the items will be retained, it will immediately, without performing any recursive calls, return a lazy sequence (a function enclosing all of the state of the call to remove-if). As we start to actually consume this sequence it will evaluate its body which will return a sequence where first is the first item in the list and rest is another lazy sequence (function) which has not been evaluated. Each call to remove-if will immediately return without performing a recursive call.

I think that one of the things that can be confusing here is that when you execute this code in the REPL it will print the resulting list. This may cause you to think that the list is actually being created when you call the function. In reality, the REPL is forcing the result list to be actualized in the process of printing it.

Hopefully this information will be helpful to anyone who is struggling with lazy-seq, especially when trying to understand how it relates to recursion.

Using the Clojure -> Macro

2009-07-14T09:54:00.001-07:00

Today I started going through the Clojure api reviewing everything up to the letter "a". Everything in this section was straight forward except for this one


(-> x form)
(-> x form & more)

Threads the expr through the forms. Inserts x as the second 
item in the first form, making a list of it if it is not a 
list already. If there are more forms, inserts the first form 
as the second item in second form, etc.

I had seen this used in the ants.clj example and did not understand it. The description here also did not make it clear (because of my inexperience with Clojure) so I did an experiment in the REPL:


user=> (-> 2 (* 2))
4
user=> (-> 2 (* 2) (+ 3))
7

As you can see from the second example above the expr 2 is passed as the second item to the first form (* 2) which results in evaluating (* 2 2) which results in 4. The result is then passed as the second item to (+ 3) etc.

To see the advantage of this, what would we have to do if we didn't have this marco.


user=> (* 2 2)
4
user=> (+ (* 2 2) 3)
7

In the first example you would not use -> because the normal way of doing this is clearer. In the second example you can start to see the benefit. These are very simple examples and perhaps arithmetic is not the best way to show its use. Let's look at two slightly more complex examples.

In the ants.clj example, some code that uses this macro is


(defn place [[x y]]
   (-> world (nth x) (nth y)))

Here we are trying to get a cell from the world which is a 2d vector. How would this look without ->?


(defn place [[x y]]
   (nth (nth world x) y))

Another good example involves getting data from nested maps. Suppose you have a map that looks like this


(def m {:type "Car" 
        :owner {:name "John" :age 27 
                :parents {:mother {:name "Jane" :age 53} 
                          :father {:name "William" :age 56}}}})

If you want to get the name of the father of the car's owner you have to work your way from the inside out.


(:name (:father (:parents (:owner m))))

Or you can use -> to work in the other direction.


(-> m :owner :parents :father :name)

To me, the second version is much easier to write.

Hopefully this clears up your understanding of the -> macro and also gives an example of the power of macros to allow you to improve the clarity of your code (please also note that the abuse of macros can greatly obscure your code).

Learning Clojure

2009-07-14T09:16:00.000-07:00

I have recently decided to focus my attention on learning Clojure. Over the past eight years most of my code has been written in Java. Recently I have started to use Groovy quite a bit. After spending the last year researching several languages including Groovy, Erlang, Ruby, Scala and Clojure I have decided that I want to invest my valuable time and energy into Clojure. In a future blog post I will give my reasoning behind this decision.

My plan of attack is to first read the "Programming Clojure" book, which I have done. I will then attempt to become more familiar with functional programming in general by reading "On Lisp", "Practical Common Lisp" and "Real World Haskell". Next I want to go through the Clojure web site and familiarize myself with the api. Finally (because you can't learn without writing code) I will try to port the core part of one of my major projects to Clojure as a learning experience. If it goes well then I will put it into production. If not then I will have learned many valuable lessons.