A “build system” is one of the most important tools in a developer’s toolbox. Roughly, it figures out how to create runnable programs from a bunch of different source files by calling out to the compiler, setting up and executing test suites, and so on. Because you interact with it daily, above all it has to be fast – but it also has to be flexible.

Around 2012 we were growing dissatisfied with OMake, then one of OCaml’s standard build systems, and decided to build our own; we called this new system Jenga. It worked quite well for us, and we thought the broader community might find it useful. So we decided to release Jenga. We hoped that when other people tried it, they’d like it, and maybe even contribute back to it. Releasing it would also make it easier for us to open source our code.

Ha! What actually happened is that nobody really wanted to use Jenga. For one thing it didn’t work on Windows. But also, to adopt Jenga was in effect to adopt the whole “Jane Street way” of building OCaml. The adoption of Jenga by those we hoped to embrace it was weak enough that we actually decided to un-open source it. And so we were back to the same place as before.

By 2016 we had had enough of this, and decided to make a simple cross-platform tool, called Jbuilder, that would allow external users to build our code without having to adopt Jenga in full, and would release us from the obligation of rewriting our builds in OCamlbuild, then an emerging standard for building OCaml projects.

Jbuilder understood the jbuild files that Jenga used for build configuration and simply executed all required compilation commands in a topological order. It wasn’t a build system in the usual sense: it would simply re-execute all commands every time (instead of only re-executing commands whose inputs have changed).

Then something strange happened. People loved Jbuilder. They started using it to build not just our packages but their own, too. At first we didn’t really understand this. Jbuilder wasn’t a real build system, after all. It was just meant to be a little compatibility shim.

What we realized, eventually, was that the compelling feature was speed. It turned out that Jbuilder was really a lot faster than the other options, compiling our projects something like 5x faster than OCamlbuild. That, plus the system being portable and easy to hack on, were the things that mattered for early adopters and contributors.

So, in collaboration with OCaml Labs (and today, Tarides), we started working on making Jbuilder into more of a real build system, adding more of the features that would be required for it to be a useful tool for the broader open-source world.

And then, we ran into another problem. The name.

It turned out there was already a Borland Java IDE called “JBuilder.” The system was long defunct, and we even went to the trouble of finding the current owners of the copyright and asking them if they’d mind us using the name. But, no dice.

So we decided to pick a new name. We did a bit of community outreach, and “Dune” emerged as the winning name.

In the meantime, Dune’s popularity had exploded. People really began using it in earnest, and we found ourselves in a somewhat ridiculous, self-inflicted situation: we now had two full build systems to maintain and support.

Jenga vs. Dune

It became clear to us that Dune was the better system—a rethought design, faster for most people’s builds, with wider adoption, and a better API and user experience—which brought up the question “when will we migrate Jane Street onto Dune?” Considering the provenance of the tool this felt like an absurd question, but oh well, that’s where we’d ended up.

The answer was inevitably “next year,” since the build systems team had plenty to work on simply keeping up with our growing codebase. (In 2016, when Dune started, we had 4M lines of OCaml code; today we have 65M, plus 5M lines of Python.) A migration to Dune was daunting enough that we never quite fully embarked on the mission. But it wasn’t so daunting to keep us from estimating, aspirationally, that it might well happen in the next six to twelve months.

It was only last year that we decided to finally rip the band-aid off. We’d grown the build systems team to five full-time engineers, so we felt that we finally had the strength to tackle this monster we’d been afraid of for so long.

Dune subsumes Jenga within Jane Street

One large chunk of work we didn’t really appreciate in the beginning was just making Dune scale to our huge codebase. Dune was quite fast externally—but that’s in part because most users built relatively tiny things compared to Jane Street’s 70M-line repository.

And Jenga hadn’t stood still, either. Over a decade, we had improved the implementation to deal with the growth of our codebase. That produced a system that was pretty well-optimized for scale, and carefully tailored to our monorepo’s requirements. Now, much of that good optimization work had to be translated to Dune.

There were more mundane problems. The build system is called by a variety of different workflows, notably, from three different editors—Vim, Emacs, and VSCode. Each one sadly had its own custom integrations with Jenga that had to be migrated to use Dune one by one.

But after more than a year of focused work we’re finally done: our codebase is now built by Dune. At the time of the switch, Dune’s performance was across the board as good or better than Jenga’s, and much better in some cases. In particular, builds where most of the build work is already in the cache (which is a surprisingly common case!) have gotten 2-3x faster.

A lot of what we’ve done to improve Dune’s performance will get open sourced, and some already has. We’re keen not to end up with two build systems again—a Jane Street fork of Dune and the external Dune—so we’re putting a lot of thought and energy into upstreaming our changes where we can.

Dune is a very good foundation for doing new things. Some of that is because Dune’s codebase is simpler and easier to work with; some because we can just focus on one system. But, features like distributed builds, shallow builds (also known as “builds without the bytes”), and cached loading of the build graph itself are all closer in reach than ever.

The exciting thing going forward is that we have a single system and our velocity in improving it is going way up. The team has also grown to 12 full-time engineers, in New York, London and Singapore, which means we’re now working on Dune 24 hours a day.

It’s been a long and sometimes meandering path, and certainly hasn’t unfolded the way we might have planned. But we think that the end result is good for OCaml’s build-system story, both within and beyond Jane Street’s walls.