My goals while writing the program were manifold (pun intended):

1. First and foremost, I wanted to understand the simulation techniques used by a research group that I collaborate with. Among these are parallel tempering (a.k.a. replica exchange) Monte Carlo, WHAM, etc.
2. I was also looking forward to learning OpenMP and the GNU Scientific Library for quite some time, so this gave me the perfect opportunity to do so.
3. Many projects have been migrating from GNU autotools to CMake. I was curious about it and wanted to become acquainted with CMake too.

The code can still be improved upon, of course. But besides cleanups or feature additions, and considering that my new toycomputer at work has two NVIDIA Tesla M2050s, I think the next step is to develop GPU capabilities.

Oh, I almost forgot that its Git repository is cloned at Github.

I was told it was trivial to roll your own doctest suite in Common Lisp. I have done that and it is easy indeed but I have included the code in incf-cl for those who would like to use it right away.

Here’s an example:

  (defpackage :test
    (:use :common-lisp :incf-cl)
    (:export :factorial))

  (in-package :test)

  (defun factorial (n &optional (acc 1))
    "Returns the factorial of N, where N is an integer >= 0.

    Examples:

    TEST> (assemble (factorial n) (<- n (range 1 5)))
    (1 2 6 24 120)

    TEST> (factorial 450/15)
    265252859812191058636308480000000

    TEST> (signals-p arithmetic-error (factorial -1))
    T

    TEST> (signals-p type-error (factorial 30.1))
    T

    TEST> (factorial 0)
    1"
    (declare (type integer n))

    (cond
      ((minusp n) (error 'arithmetic-error))
      ((/= n (floor n)) (error 'type-error)))

    (if (= n 0)
        acc
        (factorial (1- n) (* n acc))))

CL-USER> (doctest :test)
.....T

I wanted to write this myself for a long time but what prevented me from doing so was the fact that I had come up with a trivial shell/awk script that achieved almost the same goal. Since that little program was useful for someone other than me, I thought I’d polish it slightly and post it here.

This tool’s original intent was to print a call graph from a given program. It works by taking control of GDB and automatically setting breakpoints at each function call. Of course, the executable must include debugging symbols (that is, it must have been compiled with gcc’s -g option). Here’s some sample output (the format is: caller callee arguments):

xmltree_parse xmltree_itor_next (self=0x80571e8)
xmltree_itor_next xmlnode_stack_push (stack=0x80571f0, n=0x8056780)
xmltree_parse h_requires (n=0x8056808, u=0xbff9d504)
h_requires xmlnode_get_attr (self=0x8056808, name=0x80520ea "plugin")
h_requires probe_get_name (self=0x8056af0)
h_requires htab_lookup (self=0x8058100, key=0x8056868, len=3, create=0, value=0x0)


You can use the included cg2dot.awk script to produce files suitable for processing with graphviz.

Having list comprehensions in Lisp was something I was missing from Python and Haskell. So I tried to find something similar and discovered that the LOOP facility can be used as a means of achieving this goal but this, although nice, didn’t feel natural enough. In the end, I read the paper Simple and Efficient Compilation of List Comprehension in Common Lisp by Mario Latendresse which appeared at ILC 2007 and have decided to adopt that implementation for translating list comprehensions into loops.

For example, the set  \(\left\{ (x, y) \in \{0, 1, 2\}^2 \, | \, x +y = 2 \right\}\) can be expressed as

(lc (cons x y) (<- x '(0 1 2)) (<- y '(0 1 2)) (= (+ x y) 2))  ==>
((0 . 2) (1 . 1) (2 . 0))

Another example:

(lc (sin x) (<- x (range 0 .1 (/ pi 2)))) ==>
(0.0 0.09983342 0.19866933 0.29552022 0.38941833 0.47942555 0.5646425 0.6442177
 0.71735615 0.783327 0.8414711 0.8912074 0.93203914 0.96355826 0.9854498
 0.997495)

The macro I’m using is lc:

(defmacro lc (collection-form &rest quantifiers)
  "Assembles a multiset containing the results of evaluating
COLLECTION-FORM and subject to QUANTIFIERS."
  (labels ((translate (collection-form qs)
             (if (null qs)
                 `(collect ,collection-form)
                 (translate-generator-or-filter collection-form qs)))
           (translate-generator-or-filter (collection-form qs)
             (let ((q (first qs))
                   (qr (rest qs)))
               (if (eq (first q) '<-)
                   (translate-generator collection-form (second q) (third q) qr)
                   (translate-filter collection-form q qr))))
           (translate-generator (collection-form variable collection qs)
             `(nconc (loop for ,variable in ,collection
                          ,@(translate collection-form qs))))
           (translate-filter (collection-form filter-form qs)
             `(when ,filter-form ,@(translate collection-form qs))))
    (when quantifiers
      `(loop repeat 1 ,@(translate collection-form quantifiers)))))

and the helper function range is:

(defun range (a b &optional c)
  "Builds a range of numbers as Matlab would do"
  (flet ((|:| (start step stop)
           (assert (and (<= start stop) (> step 0)))
           (loop for x from start to stop by step collect x)))
    (if c
        (|:| a b c)
        (|:| a 1 b))))

A possible enhancement would be to translate to series instead
of loop in order to have lazy list comprehensions.

This code (and more) can be found in the (incf cl) utilities.

There are many improvements waiting in the pipeline but the basic
functionality is there.

You can read more about cl-buchberger here.

With Tailor you can:

1. Create a local repository using your favorite source control system of a project managed by another source control system.
2. Do your work.
3. Export your changes back to the original repository.

Here’s an example of using Tailor to import a branch of Axiom (uses Subversion) into a local darcs repository:

First of all, we create a configuration file for the project

$ tailor –verbose –source-kind svn –target-kind darcs \
–source-repository https://svn.sourceforge.net/svnroot/axiom \
–source-module branches/build-improvements \
–start-revision INITIAL \
–target-repository file:///home/rwx/lab/math/axiom/axiom-darcs/ \
–target-module axiom-build-improvements axiom-build-improvements \
> axiom-build-improvements.tailor

Now that we have the config file stored in axiom-build-improvements.tailor, we can launch the tool to do the initial import

$ tailor –configfile=axiom-build-improvements.tailor

After a while, we have a local darcs repository for that branch and we can write:

$ darcs get ~/lab/math/axiom/axiom-darcs

to get a working copy in which to do your modifications.

Today, there are a myriad of SCM tools available and projects like Tailor will become increasingly important because they let you stick to your tools of choice. In this way, instead of learning the details of other source control systems, you can focus on what you do best: writing code.

Halberd has been tested in real world scenarios for quite some time and it seems to be solid. I hope the wider audience it is gaining now will uncover some bugs and after fixing those I’ll think of it as stable software. Future work could happen in the following areas:

• Clustering algorithm
  • The module Halberd.clues.analysis currently implements an ad-hoc hierarchical clustering algorithm to isolate possible real servers. I would like halberd to report to the user the degree of trust he should place in its conclusions.
  • I think the way to go would be to test some algorithms in R (fuzzy clustering comes to mind) using real world data and see what works best before implementing anything.
• SSL session reuse
  • When an SSL/TLS session begins, the server issues an SSL session ID to the client. This ID will be used to resume transactions between client and server (remember the stateless nature of HTTP).
  • Some load balancers can keep track of which real server dealt with which SSL session and direct the client to the right server (the one having the client’s session ID in its cache). This could be used by halberd as an extra technique to enumerate real servers.
• Test suite improvements
  • The test harness is tied to my own development environment. This should change.
  • More tests never hurt.