Better Clojure Test Results with Deview

Over the past couple of months I have been doing a lot of Clojure development. During this time there were two things that have stood out as workflow killers.

The first is caused by a great feature of Clojure. Because Clojure has a solid notion of equality, you may easily test arbitrarily large Clojure data structures for equality. The problem comes when they are not equal and you need to find out what is different. Like many of you, I use clojure.test. When a test fails, the actual and expected forms are printed. Finding the difference in two large forms can be a nightmare. To feel this pain, try to find the difference in the forms below.

actual: (not (= {:remote-addr "0:0:0:0:0:0:0:1%0", :scheme :http, :query-params {}, :session {:sandbar.stateful-session/session {:project {:host "localhost", :name "deview/deview-server", :port 9000, :_id 1, :_v 1, :_type :projects}}}, :form-params {}, :request-method :get, :query-string nil, :route-params {"*" "/deview/deview-server"}, :content-type nil, :cookies {"ring-session" {:value "0bcoa4e4-a852-4976-b2e9-a697a48f1ed6"}}, :server-name "localhost", :params {"*" "/deview/deview-server"}} {:remote-addr "0:0:0:0:0:0:0:1%0", :scheme :http, :query-params {}, :session {:sandbar.stateful-session/session {:project {:host "localhost", :name "deview/deview-server", :port 9000, :_id 1, :_v 1, :_type :projects}}}, :form-params {}, :request-method :get, :query-string nil, :route-params {"*" "/deview/deview-server"}, :content-type nil, :cookies {"ring-session" {:value "0bcca4e4-a852-4976-b2e9-a697a48f1ed6"}}, :uri "/stats/deview/deview-server", :server-name "localhost", :params {"*" "/deview/deview-server"}}))

On several occasions I spent so much time looking for differences that I forgot what I was doing before encountering the problem.

Another related issue is encountered when an exception is thrown during compile time or while running a test. Usually several screens of stacktrace are output; I usually only need to see the first line. It takes up a lot of time to switch focus to the terminal window and then scroll up to find that one line that I care about. If I have encountered exceptions on the last few test runs then I also have to be careful not to scroll too far and view the results from a previous test.

These two tasks take up much more than just time. It is frustrating when you are working to solve one problem and in the process need to stop and solve a different problem. Some call this yak shaving. They also have something in common; for both of them, I am trying to manually find something that the computer should be able to find for me.

I decided to solve this problem before moving forward on any of my current projects.

Building Blocks

Smart people have already been working on these problems.

At the June Bay Area Clojure Meetup, George Jahad presented a small library he had created to solve the first problem. The library is named difform. It has two useful functions difform and sort-form: when difform is passed two forms it will display the diff, sort-form will sort a Clojure form so that it can be diffed. This is good but I want to see that diff in my test results without having to copy and paste. When a test fails, I want to immediately see what the problem is without interrupting my train of thought.

Mark McGranaghan’s clj-stacktrace makes stacktraces much better. Using the stacktrace middleware provided with Ring is a big timesaver when working on Ring web applications. It would be great if I could use this on my test result stacktraces for all my projects.

Deview

Deview is what I came up with. It makes use of the above two libraries as well as some great features of clojure.test which make it easy to capture reporting information while tests are running.

Deview runs yours tests and reports test failures and successes. If an exception is thrown, clj-stacktrace is used to clean up the stacktrace. Deview will then filter it down to show relevant (according to me) trace elements. For example:

When a test fails, the difference between the expected and actual results are displayed. The diff below shows the changes in the two forms at the beginning of this post.

I have been using this for a couple of weeks and have found it to be a huge time saver.

For complete instructions on installing and using deview, see the project README.

Deview needs to be able to run your tests, so it has to run with your project’s classpath which contains all of your project’s dependencies. Current features of deview as well as some that are planned for the future, require deview to have quite a few dependencies of its own. In order to mitigate dependency conflicts, I decided on a client/server architecture where the server is small with few dependencies and the client can have any number of dependencies.

Deview also stores information about your projects. All of this information is stored with the client so as not to corrupt your project directory.

The server is the part that you add to your project. It only depends on Clojure and clj-stacktrace. The server will run in its own process with your project’s classpath. It will run your tests, read your project.clj file and read source files. It will not in any way modify anything in your project.

The client is a Compojure web application. It can be cloned from GitHub and run from the REPL.

Features

The current focus is on testing. Once you are up and running you will be able to see a list of all of your test namespaces. You may choose to run tests for an individual namespace or for all of them.

While tests are running, your view will be updated once per second showing the current running namespace and test.

If exceptions or failures occur, you will see a short stack trace or a diff as shown in a previous section.

If you choose to run all tests and experience failures in one namespace you may easily click on that namespace to re-run its tests.

To re-run the last set of tests you simply refresh the page. This allows you to easily re-run tests with a couple of keystrokes (Command-TAB, Command-R on a mac).

The Future

There are many ways that deview can be improved in the future. Some of them are listed here.

1. Set up a time interval so that tests are automatically run every now and again
   
2. Group tests
   
3. Run groups in parallel
   
4. Add some useful metrics other than lines of code
   
5. Browse and search sources
   
6. Add links to source in test error messages
   
7. Create test configurations where bindings can be applied, for example: set a flag to not run the tests that hit the database
   
8. When tests are running automatically; when a test fails in a namespace, only run the tests for that namespace until the issue is fixed then go back to running all tests.
   
9. Find all usages of a function in one or more projects
   
10. Interface with a good code coverage tool once one emerges
    

I have more ideas than I have time. If you find deview useful and would like to help make it better then I would love the help. For more information about deview see the project README.