CheckL

Why write programs in Common Lisp but tests like Java?

My workflow for writing Common Lisp tends to be like this:

Testing is already inherent in this process, all we need is a little bit of Common Lisp magic to take advantage of it. Thus, CheckL:

(defun foo ()
  (+ 1 1)
)


(check () (foo)) ;; => 2

(defun foo ()
  (+ 1 2)
)


(check () (foo))

;     |
;     v
;
; Result 0 has changed: 3
; Previous result: 2
;    [Condition of type CHECKL::RESULT-ERROR]
;
; Restarts:
;  0: [USE-NEW-VALUE] The new value is correct, use it from now on.
;  1: [SKIP-TEST] Skip this, leaving the old value, but continue testing
;  2: [RETRY] Retry SLIME interactive evaluation request.
;  3: [*ABORT] Return to SLIME's top level.
;  4: [TERMINATE-THREAD] Terminate this thread (#<THREAD "worker" RUNNING {100586AB13}>)

Usage

Tests

Presumably you already write code to test. Possibly you even write something like this, evaluating it and manually checking the result:

(progn
  (function-1 ...)
  (function-2 ...)
)

With CheckL, you don't really have to change much. Your PROGN becomes a CHECK () and you run it. CheckL notifies you if something changes!

Results are compared with CHECKL:RESULT-EQUALP. This defaults to CL:EQUALP. Defining it for other things may be useful.

For very long values, it may be helpful to print them:

(check (:output-p t) (some-very-long-result)) => ...

If you make changes to the test, it becomes another test:

(defun foo () (+ 1 1))
(check () (foo))      ;; => 2
(check () (1- (foo))) ;; => 1

However, if you name it before you change it, it'll always compare against the same list:

(check () (foo)) ;; => 2

(defun foo () (+ 1 3))

(check (:name :two) (foo)) ;; => Error!

In this case, the old "anonymous" test (actually identified by the body of the test) is now named :two. Making changes to the test will alter the same test, now named :two, and compare against prior results.

Finally, you might want to check more than one thing in a single CHECK. You can do this with RESULTS (which is similar but not identical to VALUES; see below):

(defun foo () (+ 1 1)
(defun bar () (- 1 1))

(check (:name :two)
  (results (foo) (bar))
)
;; => 2, 0

(defun bar () (foo))

(check (:name :two)
  (results (foo) (bar))
)
;; => Error!

Or, if you want to run one or more tests:

(run :two ...)

Results

RESULTS is much like VALUES; in fact, you can use VALUES instead of RESULTS. However, RESULTS calls RESULT-TRANSLATE on each form as it occurs. By default, this copies structures and sequences, and marshals/unmarshals standard-objects.

With VALUES, the following will likely not be what you want:

(defstruct thing (x 0))

(defun incr-thing (thing)
  (incf (thing-x thing))
  thing
)


(check (:name :incr-thing)
  (let ((thing (make-thing)))
    (values
      thing
      (incr-thing thing)
      (incr-thing thing)
)
)
)


      ;; => #S(.. 2), #S(.. 2), #S(.. 2)

However, with RESULTS, we get a copy each time:

(check (:name :incr-thing)
  (let ((thing (make-thing)))
    (results
      thing
      (incr-thing thing)
      (incr-thing thing)
)
)
)


      ;; => #S(.. 0), #S(.. 1), #S(.. 2)

Additionally, it can be very useful to override RESULT-TRANSLATE, especially for complex objects:

(defmethod checkl:result-translate ((thing thing))
  (thing-x thing)
)


(run :incr-thing) ;; => 0, 1, 2

This can be useful for checking selected values of deeply-nested structures and objects.

standard-object

Another option for standard-object instances is to implement ms:class-persistant-slots (sic). RESULT-TRANSLATE calls marshal/unmarshal on objects by default, which makes a copy, if this method is defined.

This method must merely return a list of slot names, and is trivial to implement. See the documentation for cl-marshal for more details.

Categories

So you've been writing a bunch of little tests and want to run them all and see if anything has changed:

(run-all)

Easy! And you haven't had to specifically declare it so in three places. However maybe you want a bit more structure and split up your tests when you run them all. Thus categories:

(check (:name :foo :category :some-category) ...)

(run-all :some-category ...)

The default category is perhaps unsurprisingly called :default. That's pretty much all there is to categories.

Remembering results

Since we're not manually defining the result, it would be unfortunate if we happened to quit our lisp while our code still had a bug, and then weren't sure what it was. Easy enough:

(checkl-store "some-file")
;;; - later -
(checkl-load "some-file")

This uses cl-marshal and WRITE to write values to the file (overwriting it entirely). *PRINT-READABLY* is forced to t, but you can otherwise customize the output as per WRITE.

Along with revision control, it should be easy to keep track of test results and make modifications.

Formalizing

This is not meant to be a complete replacement for test suites such as FiveAM, but more of a "deformalization".

But once you've worked on a bit of code and have your buffer cluttered with (check (...) ...) forms, you probably don't want to rewrite them all as FiveAM constructs. It'd be nice if you could just sortof integrate them all with minimal effort, like this:

;; 5am doesn't have a find-suite, so you have to do this:
(defsuite :default)

(check-formal (:name :two) (foo))

Well, assuming you had FiveAM loaded before you loaded CheckL, this is exactly what you do. (I didn't want FiveAM to be a strict dependency.) Now you can do one of these, and they still do what they should:

(5am:run! :default) ;; => Pretty dots, one per VALUE
(run-all :default)  ;; => 2

We've gone from very informal testing to having things in FiveAM with minimal effort!

Note that you can still eval the CHECK-FORMAL block in your buffer and it behaves just like a CHECK block, returning its values and signaling a condition if they change.

ASDF

Wouldn't it be nice if ASDF loaded your saved CheckL values, and let you call your newly-created FiveAM tests with minimal effort? Of course!

(cl:eval-when (:load-toplevel :execute)
  (asdf:load-system :fiveam)
  (asdf:load-system :checkl)
)


(defsystem :my-system
  :description "A brand new system"
  ...
)


(defsystem :my-system-tests
  :description "A system that tests"
  
  :depends-on (:fiveam :checkl)
  :serial t
  :pathname "t"

  :components ((:file "package")
               (checkl:tests "some-test")
               (checkl:test-values "test-values"
                                   :package :my-system-tests
)
)
)


(checkl:define-test-op :my-system :my-system-test)
(checkl:define-test-op :my-system-test)

That's all! No long PERFORM definitions. Just make sure to have the EVAL-WHEN at the top. Now you can do this:

(asdf:load-system :my-system)
(asdf:test-system :my-system) ;; => (5am:run! :default)

Things of note:

Etc

I've been using this a bit and have added some conveniences. There could certainly be more. Suggestions welcome, theoretical or otherwise.

Reference: CHECKL

Functions

CHECKL-LOAD
(&OPTIONAL FILESPEC)

Load package test results from FILESPEC

CHECKL-STORE
(&OPTIONAL FILESPEC)

Store package test results to FILESPEC

CLEAR
(&REST NAMES)

Clear the test results from the tests NAMES. For clearing anonymous test results, see CLEAR-ANONYMOUS.

CLEAR-ANONYMOUS
()

Clear anonymous test results. For clearing named tests, see CLEAR.

RESULT-EQUALP
(O1 O2)

Generic function to compare test results O1 to O2. Defaults to EQUALP.

RESULT-TRANSLATE
(RESULT)

RESULT-TRANSLATE is called on RESULT before calling RESULT-EQUALP and before storing RESULT. This defaults to RESULT, but may be useful for converting more complex objects into simpler objects. For sequences and structures, COPY-SEQ and COPY-STRUCTURE are called. For STANDARD-OBJECT instances, (MS:UNMARSHAL (MS:MARSHAL OBJECT)) is called.

RUN
(&REST NAMES)
=> test-results

Run tests named NAMES, collecting their results.

RUN-ALL
(&OPTIONAL (CATEGORY :DEFAULT) &REST CATEGORIES)
=> test-results

Run all tests, optionally specifying categories.

Macros

CHECK
((&KEY NAME (CATEGORY :DEFAULT) (OUTPUT-P NIL)) &BODY BODY)
=> test-results

Run BODY. Check resulting values against the last run using CHECKL:RESULT-EQUALP, or store them if this is the first run. Sameness of the test is determined by comparing the body with EQUALP, or by NAME.

NAME may be specified to name a test. If the test exists but is anonymous (i.e., NAME has not been specified), specifying NAME will name the test and it will no longer be anonymous.

CATEGORY may be specified for running groups of tests.

If OUTPUT-P is t, the results will be printed to *standard-output* as well as returned. This may be helpful if the results are too long to see in your emacs minibuffer.

CHECK-OUTPUT
(&BODY BODY)

Use this within a CHECK block. Rebind *standard-output* and *error-output* and return a CHECK-able result.

RESULTS
(&REST VALUES)
=> VALUES

This will evaluate each subform in order and call RESULT-TRANSLATE on the result. This is especially useful if subforms repeatedly modify and return the object, e.g. (results (incf *x*) (incf *x*))