This is the third in a series of posts about the design of Iron, our new code review and release management tool. The other two posts are here and here.

Jane Street’s code is (mostly) organized as one big hg repo. Whether this is a reasonable approach is a matter of some debate. Facebook, for example, does something similar, and when they tried to work with the git community to try to make git handle their use-case, they were pretty roundly condemned. After all, who would be crazy enough to put millions of lines of fast-changing code into a single repo? Why not mint some decent modularity boundaries and use them to break things up?

I’m all for modularity, but I agree with Facebook on this one. Keeping things in a single repo has advantages even when your code is modular. In particular, it lets you use a single hash to fully describe the code in your system. That simplifies testing and deployment by making it easier to specify what needs to be tested and deployed. And it stops you from having to figure out which versions of each of your dependencies you need to build your system. Look ma, no SAT solver!

The one-big-repo approach also simplifies the process of making big, cross-cutting changes, since it gives you a way of making an atomic change that crosses a big subset of your code. This makes it easier to modernize your code and undo mistakes of the past.

Linear development and its discontents

The single-big-repo model often comes along with a (mostly) linear release process. In such a release process, there is a release branch with a single head, and changes are reviewed and then released by applying them as a single changeset to this head.

(Note that by release I’m talking about the process by which a change enters into the source tree and is this made available to other developers. This is distinct from rolling software into production, which is how you make the resulting software available to users.)

A linear release process is pleasantly simple and has a lot of advantages, but it has its limitations too. Complex changes that require significant time to review and refine can be a particular pain point.

Developing a complex change as a single patch isn’t ideal. Reading and understanding a big patch is a lot harder then reviewing a sequenced of smaller patches, at least if they’ve been broken up into pieces that have some conceptual unity.

But breaking a large change into small, independently reviewed chunks is problematic too, because the individual chunks may leave you in an awkward state where you’re not ready to roll the resulting code. This violates the desirable property that the tip of your release branch should be at or close to a rollable state.

Some projects deal with this by encouraging developers to release their incremental changes before they’re truly ready for production, but to disable them using so-called feature toggles. For us, this is a complete non-starter. Introducing untrusted code paths into a production system in the hopes that they won’t be used is too terrifying to contemplate. But when using a linear release process, it’s not clear what other choices you have.

Hierarchical features

In Iron, we use hierarchical features as a structured way of managing a non-linear release process. Hierarchical features are quite new for us, and so we’re still learning our way around them, but so far they seem to add quite a bit of flexibility, without sacrificing our ability to reason about and manage code review.

The basic idea is simple. Every repository has a single root feature, and every other feature is created as the child of an existing feature (possibly the root feature). Here, for example, is the current sub-tree corresponding to the development of Iron itself.

[21:50:31 ~]$ fe list jane/fe -display-ascii
|------------------------------------------------------|
| feature                                      | lines |
|----------------------------------------------+-------|
| jane                                         |       |
|   fe                                         | 26    |
|     backups                                  | 340   |
|     color-enabled-features-in-fe-list        | 113   |
|     comments-and-formatting                  | 1     |
|     disallow-archiving-permanent-features    | 13    |
|     do-not-show-jane-from-cr-CR-soons        | 5     |
|     fact                                     | 979   |
|     fix-int-alignment                        | 14    |
|     fix-tests                                | 24    |
|     improve-fe-session                       | 24    |
|     improve-release                          | 20    |
|     improve-review-not-enabled-error-message | 9     |
|     not-releasable-if-pending                | 30    |
|     use-async-process                        | 41    |
|------------------------------------------------------|

There are two basic operations that are used to manage this hierarchy: rebase and release.

Releasing a feature takes the changes in that feature and propagates them to the parent feature. Consider for example the feature jane/fe, which corresponds to the most recently released version of Iron, and jane/fe/backups, which is the feature where Iron’s backup support is being developed. jane/fe/backups is releasable if:

• the base revision of jane/fe/backups is the tip of jane/fe
• the review ofjane/fe/backups is complete.
• there are no outstanding issues in the code to be resolved. Issues are tracked through specially formatted comments in the source.

When you run fe release jane/fe/backups, the tip of jane/fe is advanced to the tip of jane/fe/backups. If jane/fe/backups has no children of its own, then as a convenience the bookmark is automatically removed and the feature is archived.

Where releasing moves changes from the child to the parent, rebasing does the opposite. Rebasing is useful in the case where the parent feature has changed since the child feature was branched. For example, imagine that after jane/fe/backups was created, a different feature, jane/fe/fix-tests was released into jane/fe. That means that the tip of jane/fe was changed to incorporate the changes in fix-tests.

If you run fe rebase jane/fe/backups, it will merge the tip of jane/fe/backups with jane/fe and update jane/fe/backups to that bookmark. It also changes the base revision of jane/fe/backups to be the tip of jane/fe. The end result is that jane/fe/backups will now reflect the released changes from jane/fe/fix-tests.

Calling this operation a rebase is a little confusing, because the underlying operation on the hg graph is a merge, not a rebase. But it’s not a changeset in the hg graph that’s being rebased.

Rebase and release turn out to be the basis for a rich toolset for building different release workflows. Here are a few examples.

Dependent features

It’s often useful to develop a chain of dependent features, each feature using functionality developed by the feature before it. This can make it easier to develop bigger changes as a collection of smaller and easier to verify pieces.

As an example, let’s say I want to add a new function to the List module. I can mint a feature for that.

$ fe create jane/List.fold_until -description "
    Add fold_until to List, which is a fold with an
    explicit stopping condition"

Once I commit and push, I might want to develop another feature that uses List.fold_until. I can start that dependent feature immediately (even before List.fold_until is released) by making my next feature a child of List.fold_until.

$ fe create jane/List.fold_until/clean-up-fe-show \
    -description "
       clean up the fe show code by using List.fold_until
       instead of an exception to terminate fold"

Now, clean-up-fe-show can be developed and reviewed on its own. Note that if List.fold_until changes in the meantime, there’s no real problem. We just need to rebase clean-up-fe-show to take those changes into account. And we now have the choice of releasing List.fold_until first, or of releasing them together, by first releasing clean-up-fe-show into List.fold_until, and then releasing List.fold_until.

$ fe release jane/List.fold_until/clean-up-fe-show
$ fe release jane/List.fold_until

Localizing releases

Sometimes, it just doesn’t make sense to have one release process for your entire tree. For example, you might have a project that needs a certain amount of user-testing before you’re comfortable doing a release. That testing may uncover small changes you need to make along the way. But releasing those changes to the tip of the root feature may require you to pull in unrelated changes that would invalidate more of your testing.

With Iron, we handle this by minting a feature that corresponds to the release process for the project in question. That way, you get full control over how changes move into your feature. For developing Iron itself, releases are done out of the jane/fe feature. The owners of that feature decides which of the descendents of jane/fe get released into it. And they can rebase jane/fe whenever they’re ready to pull in the most recent changes that have hit jane.

After rebasing, they can also release the changes that were done within jane/fe into jane, so others can benefit from them. All of this is under the explicit control of the owners of jane/fe, which allows them a lot of independence, while still giving them tools to easily integrate their work with others’.

Continuous release

Doing rebases and releases by hand can be a drag, especially for a feature like jane that are released into quite often.

To deal with this, we have a continuous release process for jane that is automated by Hydra, our build-bot. When a feature is proposed for release, Hydra does the following:

• Rebases the feature against its parent
• Checks that the resulting feature builds and all tests pass
• Calls fe release, which does its own checks as described earlier, including checking that the feature is fully reviewed.

If anything fails, the submitter is emailed with the details, and Hydra will try again if there are any changes either to the feature itself or to its parent.

Given that we do releases out of features other than jane, there’s no reason not to have support continuous releases in those places too. Accordingly, Iron lets you mark any feature as a continuous release feature, which signals to Hydra that it should run the continuous release process for the feature in question.

Why Iron matters

Iron is a pretty big improvement for us. Part of this is just that the system it replaces was old and crufty and had significant performance problems. But it’s more than that. Iron does a good job of breaking down the sometimes messy process of reviewing and releasing software into simple, easy to understand pieces. Given how fundamental these processes are to the act of developing software, I think that this is a real contribution.

The software itself is going to be open-sourced in the next month or two, but I’m hopeful that the ideas behind Iron will have some value as well. I’ve been struck in the last decade of working as a professional software engineer at how little is written about these issues. Outside of organizations that have built high quality tools to solve these problems internally, I think these questions are not that widely understood.