Securing Clojure Web Applications with Sandbar - Part 2

In part 1 of this series, a simple authorization scheme was added to a small Clojure web application. Continuing with this example, this post will demonstrate how features of sandbar.auth may be used to add form-based authentication and channel security.

Form-based authentication and channel security

If you would like to follow along:

$ git clone git://github.com/brentonashworth/sandbar-examples.git
$ cd sandbar-examples/security
$ open src/sandbar/examples/part_two/start.clj
$ lein deps

This code is a bit different from what we ended with last time. A stylesheet has been added and the layout has been improved. The complete source for our starting point is shown below. Make sure you understand this code before moving on.

(ns sandbar.examples.part-two.start
  (:use (ring.adapter jetty)
        (ring.middleware file)
        (compojure core)
        (hiccup core page-helpers)
        (sandbar core stateful-session auth)))

(defn query [type]
  (ensure-any-role-if (= type :top-secret) #{:admin}
                      (= type :members-only) #{:member}
                      (str (name type) " data")))

(defn layout [content]
  (html
   (doctype :html4)
   [:html
    [:head
     (stylesheet "sandbar.css")
     (icon "icon.png")]
    [:body
     [:h2 "Sandbar Security Example"]
     content
     [:br]
     [:div (if-let [username (current-username)]
             [:div
              (str "You are logged in as " username ". ")
              (link-to "logout" "Logout")])]]]))

(defn data-view [title data & links]
  [:div
   [:h3 title]
   [:p data]
   (if (seq links) links [:div (link-to "home" "Home")])])

(defn home-view []
  (data-view "Home"
             (query :public)
             [:div (link-to "member" "Member Data")]
             [:div (link-to "admin" "Admin Data")]
             [:br]
             [:div (cond (any-role-granted? :admin)
                         "Hello administrator!"
                         (any-role-granted? :member)
                         "Hello member!"
                         :else "Click on one of the links above to log in.")]))

(defn member-view []
  (data-view "Member Page"
             (query :members-only)))

(defn admin-view []
  (data-view "Admin Page"
             (query :top-secret)))

(defn permission-denied-view []
 [:div
  [:h3 "Permission Denied"]
  [:div (link-to "home" "Home")]])

(defroutes my-routes
  (GET "/home*" [] (layout (home-view)))
  (GET "/member*" [] (layout (member-view)))
  (GET "/admin*" [] (layout (admin-view)))
  (GET "/logout*" [] (logout! {}))
  (GET "/permission-denied*" [] (layout (permission-denied-view)))
  (ANY "*" [] (layout (home-view))))

(defn authenticate [request]
  (let [uri (:uri request)]
    (cond (= uri "/member") {:name "joe" :roles #{:member}}
          (= uri "/admin") {:name "sue" :roles #{:admin}})))

(def app (-> my-routes
             (with-security authenticate)
             wrap-stateful-session
             (wrap-file "public")))

(defn run []
  (run-jetty (var app) {:join? false :port 8080}))

Encryption

Our application must ensure that passwords are not sent across the network in plain text. In a production environment, one would enable SSL support on the web server and purchase a legitimate SSL certificate from a valid authority for use with the site's domain. For development, it is good enough to create a self-signed certificate and turn on Jetty's SSL support.

Use Java's keytool to create a self-signed certificate.

$ keytool -genkey -alias sandbar -keyalg RSA -keystore my.keystore -keypass foobar
Enter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]: localhost
What is the name of your organizational unit?
[Unknown]: dev
What is the name of your organization?
[Unknown]: clojure
What is the name of your City or Locality?
[Unknown]: New York
What is the name of your State or Province?
[Unknown]: New York
What is the two-letter country code for this unit?
[Unknown]: US
Is CN=localhost, OU=dev, O=clojure, L=New York, ST=New York, C=US correct?
[no]: y

For this example, the password "foobar" was entered. keytool has created a keystore in the file named my.keystore. Make sure this file is located in the root directory of the security module (at the same level as the public directory).

To make use of this keystore, update the run function so that it matches the version shown below.

(defn run []
  (run-jetty (var app) {:join? false :ssl? true :port 8080 :ssl-port 8443
                        :keystore "my.keystore"
                        :key-password "foobar"}))

Setting :join? to false will cause the call to run-jetty to return so that the REPL may still be used. The :ssl-port defaults to 443; here we set it here to 8443. Everything else is straight forward. If you are following along, now would be a great time to start a REPL and test that everything is working as expected.

$ lein repl


user=> (use 'sandbar.examples.part-two.start)
user=> (run)

Navigating to https://localhost:8443/ and http://localhost:8080/ confirms that the application may be used over SSL or standard http and that everything works the same as it did before.

Note: You will get a warning message because the certificate that we are using is not legitimate. This is fine for development; do what you need to do to add an exception for this certificate.

Adding form-based authentication

In the last post, the with-security middleware was added and configured to use our authenticate function. In this section, the authenticate function will be replaced with an authentication function from sandbar.form-authentication and a pre-built login form will be added.

Start by adding the required namespaces sandbar.form-authentication and sandbar.validation.

Delete the authenticate function and replace authenticate with form-authentication in our with-security middleware. form-authentication will redirect a user to a login form when that user is not authenticated.

To implement the login form, add form-authentication-routes to the list of routes.

(form-authentication-routes (fn [_ c] (layout c))
                            (form-authentication-adapter))

The parameters to form-authentication-routes are: a layout function and something that satisfies the protocol FormAuthAdapter. The layout function must take two parameters: the request and the content to layout. The FormAuthAdapter protocol specifies two functions which allow us to adapt this component to our system. The functions are load-user and validate-password. load-user takes a username and password and returns a user map. A user map must at least have the keys :username and :roles. validate-password returns a function that can validate the user map created by load-user.

(defrecord DemoAdapter []
  FormAuthAdapter
  (load-user
   [this username password]
   (let [login {:username username :password password}]
     (cond (= username "member")
           (merge login {:roles #{:member}})
           (= username "admin")
           (merge login {:roles #{:admin}})
           :else login)))
  (validate-password
   [this]
   (fn [m]
     (if (= (:password m) (:username m))
       m
       (add-validation-error m "Username and password do not match!")))))

(defn form-authentication-adapter []
  (DemoAdapter.))

This implementation of load-user will simply look at the username to determine if the user is a member or admin. The validate-password implementation will ensure that the username and password are the same. add-validation-error is a function from sandbar.validation which is being used here to display an error message.

(For more information about validators, see the post Clojure Macros Make Me Happy. I will not go into any more detail here.)

After making these changes, saving them and reloading the namespace,

user=> (use :reload-all 'sandbar.examples.part-two.start)

we can return to our application where we should now have an operational login form. Try submitting the form while it is empty. Try entering a username and password that do not match. The form does the right thing in each situation.

Form Customization

At this point you may want to customize the field names on the form as well as the error messages that are displayed. While we are at it, let's make a custom logout landing page. Create a map with keys that correspond to the fields and errors that we would like to update as well as the key :logout-page that indicates where to go after we logout.

(def properties
     {:username "Username"
      :password "Password"
      :username-validation-error "Enter either admin or member"
      :password-validation-error "Enter a password!"
      :logout-page "/after-logout"})

Update the form-authentication-adapter constructor to merge these properties into our DemoAdapter. Because Clojure's records implement the persistent map interface, the form-authentication module uses the FormAuthAdapter as a map to look up field names and error messages for the login form.

(defn basic-auth-adapter []
  (merge (DemoAdapter.) properties))

After making this change, we will see our own field names and error messages displayed on the login form.

Next, create the logout landing page by replacing the empty map that was passed to logout! with properties and then creating a route

(GET "/after-logout" [] (layout (after-logout-view)))

and view.

(defn after-logout-view []
 [:div
  [:h3 "Logout"]
  [:p "You are no longer logged in!"]
  [:div (link-to "home" "Home")]])

In all the changes that have been made so far, a pattern is emerging; add components in the form of middleware or parametrized routes then adapt it to our project. This same pattern will be used in the next section to add channel security.

Channel security

One thing that you may have noticed is that even though SSL is enabled, there is no way to control when it is used and when not. The goal in this application is to secure passwords sent from the client to the server and, because of the additional delay, to use SSL on as few pages as possible. This may be done by adding the with-secure-channel middleware which takes four parameters: the routes to be wrapped, a security configuration, the port and the SSL port. After adding this middleware, the app var definition now looks like this:

(def app (-> my-routes
            (with-security basic-auth)
            wrap-stateful-session
            (wrap-file "public")
            (with-secure-channel security-config 8080 8443)))

The security configuration security-config is a vector of pairs. Each pair is a regular expression literal followed by a configuration. We use a vector here instead of a map because each entry is checked against the current URI in order with the first match being selected.

(def security-config
     [#"/login.*" :ssl
      #".*.css|.*.png" :any-channel
      #".*" :nossl])

Three keywords are used to represent the three kinds of channel security: :ssl, :nossl and :any-channel. The above configuration causes the login screen to always be accessed through SSL, images and stylesheets to go over any channel and everything else to go through standard http.

A future post will show how this same vector may be used to authorized access based on URI patterns.

If you did not follow along, you may want to take a look at the final version of the source for this exmaple.

Conclusion

Progress has been made, but this app is still not secure. Passwords should not be hard coded into an application like this. The application may be improved by having a way to easily assign different passwords to each user and store them securely. There will be at least two more posts in this series; one them will cover some of the new features of Sandbar which can help make this easy for the most common cases. The other will demonstrate how to authorize access to resources based on URI patterns.

Clojure Protocols and the Expression Problem

The official release of Clojure 1.2 is just around the corner. One of the most significant changes in this release is the addition of protocols and datatypes. Below, we will explore one way that these features may be used to improve your Clojure programs. Specifically, we will look at how protocols may be used to avoid the Expression Problem.

According to Wikipedia, the term "Expression Problem" was coined by Philip Wadler:

“The Expression Problem is a new name for an old problem. The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety (e.g., no casts).”

We experience this problem when we want to add functionally to library code that is outside of our control. In the Java world, wrappers are commonly used to adapt a class to an interface. The problem with this approach is that the identity of the thing has changed and now our system has to deal with two types for the same thing. In the Ruby world, with its open classes, monkey patching may be used to add functions to a class. Let's face it, anything with the word monkey in it can’t be good. Monkey patching changes the class for everyone and can cause unforeseen problems. Both of these approaches add incidental complexity to our code which is the exact thing that we are trying to avoid by using Clojure.

In this post we will go over some working code that is intended to simulate a real life situation where we may use protocols to avoid the Expression Problem. The code is located in the protocol-examples project in the module named expresssion-problem. If you would like to follow along:

$ git clone git://github.com/brentonashworth/protocol-examples.git
$ cd protocol-examples/expression-problem
$ lein deps && lein javac && lein test com.corp.employee-test
$ open src/clj/com/corp/employee.clj


Simulating an integration problem

Imagine that we are creating a system to work with employees. We have created a library, shown below, which performs various payroll and benefit calculations on employee data. As is the custom of Clojure people, employees are represented as maps so that they gain all of the benefits of being "generically manipulable".

(ns com.corp.employee)

(defn- bonus [years performance]
  (* (+ (* years 500)
        1000)
     (/ performance 10)))

(defn- earned-vacation [years]
  (+ 10 (* years 2)))

(defn- vacation-value [rate years]
  (* rate (* 8 (earned-vacation years))))

(defn payroll [employee hours]
  (* hours
     (:rate employee)))

(defn total-payroll
  [coll hours-map]
  (reduce (fn [total next]
            (let [[{rate :rate} hours] (val next)]
              (+ total (* rate hours))))
          0
          (merge-with conj (group-by :name coll) hours-map)))

(defn employee-bonus [e]
  (apply bonus ((juxt :years :perf) e)))

(defn employee-vacation-value [e]
  (apply vacation-value ((juxt :rate :years) e)))

(defn total-emp-benefits [employee]
  (+ (employee-vacation-value employee)
     (employee-bonus employee)))

(defn total-benefits [coll]
  (reduce + (map total-emp-benefits coll)))

(defn make-employee
  [name years perf rate]
  {:name name :years years :perf perf :rate rate})

The problem

Now imagine that we have a new requirement. The existing payroll system will need to be able to use our new library to calculate benefits. They happen to be using Clojure as well but the payroll system is written in Java and cannot be changed. Their system has a Java class named com.company.Employee which contains the information that we need but does not conform to our interface. Specifically, they will need to be able to call total-emp-benefits and total-benefits passing instances of com.company.Employee.

We may start to solve this problem by creating some tests that demonstrate what we would like our system to do (if you are following along, these tests are already included in com.corp.integration-test).

(ns com.corp.integration-test
  (use (clojure test)
       (com.corp employee))
  (import com.company.Employee))

(def e1 (make-employee "jim" 2 8 20))
(def e2 {:name "sue" :years 15 :perf 6 :rate 60})
(def e3 (Employee. "james" 2 8 30M))
(def employees [e1 e2 e3])

(deftest test-employee
  (is (= (.calculatePayroll e3 40)
         1200M)))

(deftest test-total-emp-benefits
  (is (= (total-emp-benefits e3)
         4960M)))

(deftest test-total-benefits
  (is (= (total-benefits employees)
         33100M)))

In these tests we create two employee maps and one employee using the closed Employee class from the payroll system. Our first test is a sanity check; calling one of the methods that exists in Employee. The other two tests pass instances of Employee to functions in our system. The last two tests will not pass because our current implementation expects plain Clojure maps.

Whatever our solution is, we want to ensure that we do not change the caller's contract. There is a lot of code in production that is using our library and we don't want to force our callers to have to change their code.

What is our contract? We have six reporting functions that are designed to work with maps of employee data. We also have a constructor function that provides a more compact way to creating these maps. Our callers may or may not be using this constructor. The tests that we have in place are testing the caller's contract so if we can get all of the tests working without making any changes to them then we have been successful.

Clojure provides two solutions to this problem: multimethods and protocols. Multimethods dispatch on a function of their arguments, protocol functions dispatch on the type of their first argument. As long as we are happy with the form of dispatch, protocols provide the additional benefits of interface organization and faster dispatch. We will choose to use protocols for our solution.

Adding a protocol

We start by defining the Benefits protocol. We choose the functions from our current system that calculate benefits and take an employee as the first argument.

(defprotocol Benefits
 (employee-bonus [this])
 (employee-vacation-value [this]))

Because these functions are defined in our protocol, we can no longer use them as standalone functions. We must implement them for each datatype that we would like them to support. To start with, we want to ensure that Clojure maps continue to work. We use extend-protocol to implement the functions for maps using our existing implementations. If you are following along, add the following form and remove the functions employee-bonus and employee-vacation-value.

(extend-protocol Benefits
  clojure.lang.IPersistentMap
  (employee-bonus [this]
                  (apply bonus ((juxt :years :perf) this)))
  (employee-vacation-value [this]
                           (apply vacation-value ((juxt :rate :years) this))))

After making these changes our original tests should still work.

$ lein test com.corp.employee-test

Next, we solve the problem that we set out to solve by extending the type com.company.Employee with the Benefits protocol. The final extend-protocol form is shown below.

(extend-protocol Benefits
  
  clojure.lang.IPersistentMap
  (employee-bonus [this] (bonus (:years this)
                                (:perf this)))
  (employee-vacation-value [this] (vacation-value (:rate this)
                                                  (:years this)))

  com.company.Employee
  (employee-bonus [this] (bonus (.getYearsWithCompany this)
                                (.getCurrentPerformanceRating this)))
  (employee-vacation-value [this] (vacation-value (.getHourlyRate this)
                                                  (.getYearsWithCompany this))))

Now we may run all of our tests and... Congratulations! Everything works. From the callers perspective, we have added the functions employee-bonus and employee-vacation-value to the Employee class. Notice that we have not globally changed this class, these functions are only available within the context of the com.corp.employee namespace.

Improving performance

Because we have chosen to use protocols, we can make one final improvement to our library. We can create a record based on our protocol and then update the make-employee function to create instances of this record instead of creating maps. This will increase application performance. First we create the new record.

(defrecord StandardBenefits [name years perf rate]
  Benefits
  (employee-bonus [this] (bonus years perf))
  (employee-vacation-value [this] (vacation-value rate years)))

Next, update the make-employee function to create an instance of StandardBenefits instead of a map.

(defn make-employee [name years perf rate]
 (StandardBenefits. name years perf rate))

Run all the tests to confirm that everything works.

Notice that even the functions payroll and total-payroll continue to work when they are passed instances of StandardBenefits even thought they are not part of our protocol and are expecting plain Clojure maps. They work because defrecord provides a complete implementation of a persistent map which is the main advantage of using it over deftype.

If you were not following along then you may want to have a look at the finished version.

Conclusion

The most important thing about what we have done here is what we didn't do. We made no change to com.company.Employee, we didn't change our reporting functions or even the functions payroll and total-payroll which depend on the persistent map abstraction, and we made no change to the caller's contract.

For more information about Clojure's protocols see the datatypes and protocols pages on the Clojure web site. There is also a very informative presentation, "Clojure 1.2 Protocols", by Stuart Halloway.

Securing Clojure Web Applications with Sandbar - Part 1

Authorization in the model

This is the first in a series of posts that will show how to use Sandbar to secure web applications. Sandbar is a web application library which adds high level abstractions to Compojure and Ring. The sandbar.auth namespace provides code to make role-based security simple.

Security can be divided into two essential parts: authentication and authorization. In this post we will only explore authorization and are therefore getting only part of the security story. In part two we will add encryption, channel security and form based authentication. In part three we will authorize based on URI patterns. Finally, in the fourth installment, we will use Sandbar to create a user administration tool.

The complete code for this example is located in the sandbar-examples repository under security. If you would like to follow along then clone that repository, get the dependencies, start a REPL and open the sandbar.examples.part-one.start namespace in your favorite editor.

$ git clone git://github.com/brentonashworth/sandbar-examples.git
$ cd sandbar-examples/security
$ open src/sandbar/examples/part_one/start.clj
$ lein deps
$ lein repl


user=> (use ‘sandbar.examples.part-one.start)
user=> (run)

We will start with a small Compojure application and, throughout this series, add features to it. The complete source for our starting point is shown below.

(ns sandbar.examples.part-one.start
 (:use (ring.adapter jetty)
       (compojure core)
       (hiccup core page-helpers)))

(defn query [type]
 (str (name type) " data"))

(defn layout [content]
 (html
  [:html
    [:body
    [:h2 "Sandbar Authorization Example"]
    content]]))

(defn data-view [title data & links]
 [:div
  [:h3 title]
  [:p data]
  (if (seq links) links [:div (link-to "home" "Home")])])

(defn home-view []
 (data-view "Home"
            (query :public)
            [:div (link-to "member" "Member Data")]
            [:div (link-to "admin" "Admin Data")]))

(defn member-view []
 (data-view "Member Page"
            (query :members-only)))

(defn admin-view []
 (data-view "Admin Page"
            (query :top-secret)))

(defroutes my-routes
 (GET "/home*" [] (layout (home-view)))
 (GET "/member*" [] (layout (member-view)))
 (GET "/admin*" [] (layout (admin-view)))
 (ANY "*" [] (layout (home-view))))

(def app my-routes)

(defn run []
 (run-jetty (var app) {:join? false :port 8080}))

(The app var is not required here but having it allows us to load this code once and then make all of the changes below without having to restart our REPL.)

This is a complete application which will show three views: the home page will show public data, the member page will show members-only data and the admin page will show top-secret data. The query function retrieves the data that is displayed.

Add sandbar.auth, middleware and create an authentication function

We would like to have two roles: administrators and members. Only administrators may see top-secret data and only members may see members-only data. We would also like our application to have different behavior based on a user's role.

Before we may add any of Sandbar's functionality, we need to first add Sandbar to our namespace declaration.

(:use (ring.adapter jetty)
       (compojure core)
       (hiccup core page-helpers)
       (sandbar stateful-session auth))

Here we have added the sandbar.stateful-session and sandbar.auth namespaces.

Next, we add some middleware.

(def app
    (-> my-routes
        (with-security authenticate)
        wrap-stateful-session))

Our routes have been wrapped with the with-security and wrap-stateful-session middleware. with-security does all of the hard work. Here it has one argument, an authentication function, which we have not written. The wrap-stateful-session middleware is used by with-security to store user information.

authenticate is where you would do whatever you need to do to figure out who the current user is based on the information in the request. This function is very simple, it takes the request map as an argument and returns a single user map or nil. The user map has two keys: :name, which is a string, and :roles, which is a set of role keywords. For this example, we will use the function below.

(defn authenticate [request]
 (let [uri (:uri request)]
   (cond (= uri "/member") {:name "joe" :roles #{:member}}
         (= uri "/admin") {:name "sue" :roles #{:admin}})))

This function will not secure our application but it does demonstrate how simple it is to add your own authentication scheme. For our example, we simply authenticate the user based on the first URI that is accessed. This function will only be called if the current user is not authenticated.

After making these changes we should still have a working application. Reload the namespace in the REPL and try it out.

user=> (use :reload-all ‘sandbar.examples.part-one.start)

There is a lot of new code running, but the functionality has not changed. At least we know we haven't broken anything.

Protecting data in the model

All of our data is still out in the open. To protect it, we could use the ensure-any-role macro. It takes two arguments: a set of roles and a form that is being protected. The protected form will only be executed if the user is authenticated and has one of the roles in the set.

(defn query [type]
 (ensure-any-role #{:member :admin}
                  (str (name type) " data")))

This isn't really what we are looking for. A simple visit to the home page will result in a java.lang.StackOverflowError. Understanding why will help us understand what is going on behind the scenes. ensure-any-role will throw an exception when the current user is not authenticated. The with-security middleware will catch the exception and then call authenticate to authenticate this user. Our authentication function will not authenticate the user becuase it only authenticates visitors to the admin and member pages. Finally, with-security will redirect the user to the home page and the cycle continues.

This is probably a bug which needs fixing, but it does indicate that we have a problem. There is no way for an unauthenticated user to view the public data on the home page. We could start using conditionals but that will get ugly. Instead, there is a better macro to use in this situation named ensure-any-role-if.

(defn query [type]
 (ensure-any-role-if (= type :top-secret) #{:admin}
                     (= type :members-only) #{:member}
                     (str (name type) " data")))

This macro takes an odd number of arguments. The final argument is the form that you are protecting. The other arguments are pairs of predicates and sets of roles. If the predicate is true then the user must be a member of one of the roles in the corresponding set. If none of the predicates are true then the protected code will be run without authentication.

We can safely try it now. If you first click on the "Member Page" link you will see the members-only data. If you then try to click on the "Admin Page" link, nothing will happen. Our data has been protected.

Logout and permission denied

When ensure-any-role or ensure-any-role-if notice that the current user is authenticated but does not have the correct role, it will redirect the user to "/permission-denied". It would be nice if we could see a better permission denied page. Add the following route and implementation of the permission-denied-view function.

(GET "/permission-denied*" [] (layout (permission-denied-view)))

(defn permission-denied-view []
 [:div
  [:h3 "Permission Denied"]
  [:div (link-to "home" "Home")]])

We may also want to logout so that we can easily try out the different roles.

(GET "/logout*" [] (logout! {}))

The logout route will use a function named logout! from sandbar.auth to clear the current user from the session. The parameter to logout! is a map which may contain the key :logout-page. If this key is not found then the user will be redirected to “/” after logout.

The user will need some kind of link to allow them to logout easily. Add the following div to the end of the body section of the layout.

[:div (if-let [username (current-username)]
        [:div
          (str "You are logged in as " username ". ")
          (link-to "logout" "Logout")])]

The current-username function is another function from sandbar-auth. It will get the current user's username from the session or return nil if one does not exist. After reloading in the REPL, we now see the username at the bottom of the page, we may logout and we have a proper permissions denied page.

Checking the role

As a final flourish for this example we will add the following div to our layout underneath the content and above the logout code that we added above.

[:div (cond (any-role-granted? :admin)
            "Hello administrator!"
            (any-role-granted? :member)
            "Hello member!"
            :else "Click on one of the links above to log in.")]

This shows how to use the any-role-granted? predicate. Here we show a different message based on the role of the current user. any-role-granted? may be passed a single role or a set of roles. It will return true if the user is a member of one of those roles.

The complete source code for the new authorized version is located in the namespace sandbar.examples.part-one.complete.

Conclusion

We haven’t actually secured anything in this example. But hopefully you can see how that would be done by creating a different implementation of authenticate. In the next installment, we will look at one way to do this and add form based authentication to this example project as well as encryption and channel security.

For more information on Sandbar, see the wiki and check out the source code. The source for sandbar.auth is only 245 lines. Sandbar is still quite new and needs a lot of work. I hope this series will draw more attention to it and help to make it better.