At Jane Street, an “expect test” is a test where you don’t manually write the output you’d like to check your code against – instead, this output is captured automatically and inserted by a tool into the testing code itself. If further runs produce different output, the test fails, and you’re presented with the diff.

For example, we might write:

let%expect_test "foo" = printf "Mean: %F" (mean 1. 4.)

When compiled we will get a corrected file with the following output:

let%expect_test "foo" =
  printf "Mean: %F" (mean 1. 4.)
  [%expect {| 2.5 |}]

Since this is correct, we accept the output and this becomes the contents of our test file. Subsequent runs of the test produce the same output, so there are no further diffs. If, however, we change the definition of mean to, say, the geometric mean, our tool generates a build error and a diff:

let%expect_test "foo" =
  printf "Mean: %F" (mean 1. 4.)
-|  [%expect {| 2.5 |}]
+|  [%expect {| 2.0 |}]

The most interesting aspect of this style is the workflow: we write the initial code, our build system computes the correct output, and our editor integration means it takes just a couple of keypresses to accept the result if we’re happy with it. Then we check it into our source code repository, at which point continuous integration systems automatically ensure our tests continue to work in the presence of every developer’s bugbear – other developers.

Expecting things from hardware

When we develop a hardware design we always do so in the presense of a testbench. A testbench provides stimulus for the design, monitors the computed outputs and ensures that they are correct.

At Jane Street, developing and testing hardware designs is performed using an Embedded Domain Specific Language called Hardcaml, which unsurprisingly is written in OCaml.

One of the key features of Hardcaml is that it provides a cycle-accurate simulator. This allows us to develop both the hardware design and testbench in OCaml.

The following is a simple 8-bit counter. It resets back to 0 when the clear signal is high, and counts up when the incr signal is high. Otherwise it holds its previous value.

open Hardcaml
open Hardcaml.Signal
open Hardcaml_waveterm

module I = struct
    type 'a t =
      { clock : 'a
      ; clear : 'a
      ; incr : 'a
      }
      [@@deriving sexp_of, hardcaml]
end

module O = struct
    type 'a t =
      { dout : 'a[@bits 8]
      }
      [@@deriving sexp_of, hardcaml]
end

let create (i : _ I.t) =
  { O.dout =
      reg_fb
        (Reg_spec.create ~clock:i.clock ~clear:i.clear ())
        ~enable:i.incr
        ~w:8
        (fun d -> d +:. 1)
  }
;;
val create : t I.t -> t O.t = <fun>

The following is a simple testbench for the counter which shows its behaviour for different values of clear and incr.

module Simulator = Cyclesim.With_interface(I)(O)

let testbench () =
  let sim = Simulator.create create in
  let inputs = Cyclesim.inputs sim in
  let outputs = Cyclesim.outputs sim in
  let step ~clear ~incr =
    inputs.clear := if clear=1 then Bits.vdd else Bits.gnd;
    inputs.incr := if incr=1 then Bits.vdd else Bits.gnd;
    Printf.printf "clear=%i incr=%i dout=%i\n"
      clear incr (Bits.to_int !(outputs.dout));
    Cyclesim.cycle sim
  in
  step ~clear:0 ~incr:0;
  step ~clear:0 ~incr:1;
  step ~clear:0 ~incr:1;
  step ~clear:1 ~incr:0;
  step ~clear:0 ~incr:0;
  step ~clear:0 ~incr:0
;;
val testbench : unit -> unit = <fun>

testbench ();;
clear=0 incr=0 dout=0
clear=0 incr=1 dout=0
clear=0 incr=1 dout=1
clear=1 incr=0 dout=2
clear=0 incr=0 dout=0
clear=0 incr=0 dout=0
- : unit = ()

We can now capture this behaviour as an expect test.

let%expect_test "counter" =
  testbench ();
  [%expect {|
    clear=0 incr=0 dout=0
    clear=0 incr=1 dout=0
    clear=0 incr=1 dout=1
    clear=1 incr=0 dout=2
    clear=0 incr=0 dout=0
    clear=0 incr=0 dout=0
  |}]
;;

Waveform expect tests

Digital waveforms are commonly used during hardware development to capture the time-varying behaviour of signals relative to one another. Using the Hardcaml_waveterm library we can print waveforms from Hardcaml simulations.

let testbench () =
  let sim = Simulator.create create in
  let waves, sim = Waveform.create sim in
  let inputs = Cyclesim.inputs sim in
  let step ~clear ~incr =
    inputs.clear := if clear=1 then Bits.vdd else Bits.gnd;
    inputs.incr := if incr=1 then Bits.vdd else Bits.gnd;
    Cyclesim.cycle sim
  in
  step ~clear:0 ~incr:0;
  step ~clear:0 ~incr:1;
  step ~clear:0 ~incr:1;
  step ~clear:1 ~incr:0;
  step ~clear:0 ~incr:0;
  step ~clear:0 ~incr:0;
  waves
;;
val testbench : unit -> Waveform.t = <fun>

let waves = testbench ();;
val waves : Waveform.t = <abstr>
Waveform.print ~display_height:12 waves;;
┌Signals────────┐┌Waves──────────────────────────────────────────────┐
│clock          ││┌───┐   ┌───┐   ┌───┐   ┌───┐   ┌───┐   ┌───┐   ┌──│
│               ││    └───┘   └───┘   └───┘   └───┘   └───┘   └───┘  │
│clear          ││                        ┌───────┐                  │
│               ││────────────────────────┘       └───────────────   │
│incr           ││        ┌───────────────┐                          │
│               ││────────┘               └───────────────────────   │
│               ││────────────────┬───────┬───────┬───────────────   │
│dout           ││ 00             │01     │02     │00                │
│               ││────────────────┴───────┴───────┴───────────────   │
│               ││                                                   │
└───────────────┘└───────────────────────────────────────────────────┘
- : unit = ()

Since they are just text, these can also be captured with expect tests:

let%expect_test "counter" =
  let waves = testbench () in
  Waveform.print ~display_height:12 waves;
  [%expect {|
┌Signals────────┐┌Waves──────────────────────────────────────────────┐
│clock          ││┌───┐   ┌───┐   ┌───┐   ┌───┐   ┌───┐   ┌───┐   ┌──│
│               ││    └───┘   └───┘   └───┘   └───┘   └───┘   └───┘  │
│clear          ││                        ┌───────┐                  │
│               ││────────────────────────┘       └───────────────   │
│incr           ││        ┌───────────────┐                          │
│               ││────────┘               └───────────────────────   │
│               ││────────────────┬───────┬───────┬───────────────   │
│dout           ││ 00             │01     │02     │00                │
│               ││────────────────┴───────┴───────┴───────────────   │
│               ││                                                   │
└───────────────┘└───────────────────────────────────────────────────┘
  |}]
;;

Expect test workflow

Using expect tests in this way makes the hardware development process a lot more similar to developing software. It allows us to leverage years’ worth of tool development and integrate with the Iron workflow that is central to development at Jane Street.

It is also simply a lot nicer to write testbenches in OCaml than in traditional RTL languages.

There are without doubt some drawbacks – embedding a waveform in source code means you can’t reasonably display that much information. This means you have to think carefully about what information to expose (some might argue this is an advantage). Also, while we do have an interactive viewer application for waveforms, it can be a bit of work to factor code so that it can work both in the interactive viewer and expect tests.

Finally, we have yet to teach our diff tools about waveforms – the current diffs are not terrible, but they could be better.

Even with these issues, using expect tests while developing hardware has become very common within our team. This gives us a good incentive to spend some time improving our tooling.

One final note: while we do all of this using our internal tooling, all of the underlying code is available on Github as open source: expect tests and hardcaml. And while the external tooling isn’t quite as good as the internal tools (yet!), Dune provides pretty good support for working with expect tests.