Jane Street’s code is (mostly) organized as one big hg repo. Whether this is a reasonable approach is a matter of some debate. Facebook, for example, does something similar, and when they tried to work with the git community to try to make git handle their use-case, they were pretty roundly condemned. After all, who would be crazy enough to put millions of lines of fast-changing code into a single repo? Why not mint some decent modularity boundaries and use them to break things up?
I’m all for modularity, but I agree with Facebook on this one. Keeping things in a single repo has advantages even when your code is modular. In particular, it lets you use a single hash to fully describe the code in your system. That simplifies testing and deployment by making it easier to specify what needs to be tested and deployed. And it stops you from having to figure out which versions of each of your dependencies you need to build your system. Look ma, no SAT solver!
The one-big-repo approach also simplifies the process of making big, cross-cutting changes, since it gives you a way of making an atomic change that crosses a big subset of your code. This makes it easier to modernize your code and undo mistakes of the past.
Linear development and its discontents
The single-big-repo model often comes along with a (mostly) linear release process. In such a release process, there is a release branch with a single head, and changes are reviewed and then released by applying them as a single changeset to this head.
(Note that by release I’m talking about the process by which a change enters into the source tree and is this made available to other developers. This is distinct from rolling software into production, which is how you make the resulting software available to users.)
A linear release process is pleasantly simple and has a lot of advantages, but it has its limitations too. Complex changes that require significant time to review and refine can be a particular pain point.
Developing a complex change as a single patch isn’t ideal. Reading and understanding a big patch is a lot harder then reviewing a sequenced of smaller patches, at least if they’ve been broken up into pieces that have some conceptual unity.
But breaking a large change into small, independently reviewed chunks is problematic too, because the individual chunks may leave you in an awkward state where you’re not ready to roll the resulting code. This violates the desirable property that the tip of your release branch should be at or close to a rollable state.
Some projects deal with this by encouraging developers to release their incremental changes before they’re truly ready for production, but to disable them using so-called feature toggles. For us, this is a complete non-starter. Introducing untrusted code paths into a production system in the hopes that they won’t be used is too terrifying to contemplate. But when using a linear release process, it’s not clear what other choices you have.
In Iron, we use hierarchical features as a structured way of managing a non-linear release process. Hierarchical features are quite new for us, and so we’re still learning our way around them, but so far they seem to add quite a bit of flexibility, without sacrificing our ability to reason about and manage code review.
The basic idea is simple. Every repository has a single root feature, and every other feature is created as the child of an existing feature (possibly the root feature). Here, for example, is the current sub-tree corresponding to the development of Iron itself.
[21:50:31 ~]$ fe list jane/fe -display-ascii |------------------------------------------------------| | feature | lines | |----------------------------------------------+-------| | jane | | | fe | 26 | | backups | 340 | | color-enabled-features-in-fe-list | 113 | | comments-and-formatting | 1 | | disallow-archiving-permanent-features | 13 | | do-not-show-jane-from-cr-CR-soons | 5 | | fact | 979 | | fix-int-alignment | 14 | | fix-tests | 24 | | improve-fe-session | 24 | | improve-release | 20 | | improve-review-not-enabled-error-message | 9 | | not-releasable-if-pending | 30 | | use-async-process | 41 | |------------------------------------------------------|
There are two basic operations that are used to manage this hierarchy: rebase and release.
Releasing a feature takes the changes in that feature and propagates them to the
parent feature. Consider for example the feature
jane/fe, which corresponds to
the most recently released version of Iron, and
jane/fe/backups, which is the
feature where Iron’s backup support is being developed.
- the base revision of
jane/fe/backupsis the tip of
- the review of
- there are no outstanding issues in the code to be resolved. Issues are tracked through specially formatted comments in the source.
When you run
fe release jane/fe/backups, the tip of
jane/fe is advanced to
the tip of
jane/fe/backups has no children of its own,
then as a convenience the bookmark is automatically removed and the feature is
Where releasing moves changes from the child to the parent, rebasing does the
opposite. Rebasing is useful in the case where the parent feature has changed
since the child feature was branched. For example, imagine that after
jane/fe/backups was created, a different feature,
jane/fe. That means that the tip of
jane/fe was changed to
incorporate the changes in
If you run
fe rebase jane/fe/backups, it will merge the tip of
jane/fe and update
jane/fe/backups to that bookmark.
It also changes the base revision of
jane/fe/backups to be the tip of
jane/fe. The end result is that
jane/fe/backups will now reflect the
released changes from
Calling this operation a rebase is a little confusing, because the underlying operation on the hg graph is a merge, not a rebase. But it’s not a changeset in the hg graph that’s being rebased.
Rebase and release turn out to be the basis for a rich toolset for building different release workflows. Here are a few examples.
It’s often useful to develop a chain of dependent features, each feature using functionality developed by the feature before it. This can make it easier to develop bigger changes as a collection of smaller and easier to verify pieces.
As an example, let’s say I want to add a new function to the
List module. I
can mint a feature for that.
$ fe create jane/List.fold_until -description " Add fold_until to List, which is a fold with an explicit stopping condition"
Once I commit and push, I might want to develop another feature that uses
List.fold_until. I can start that dependent feature immediately (even before
List.fold_until is released) by making my next feature a child of
$ fe create jane/List.fold_until/clean-up-fe-show \ -description " clean up the fe show code by using List.fold_until instead of an exception to terminate fold"
clean-up-fe-show can be developed and reviewed on its own. Note that if
List.fold_until changes in the meantime, there’s no real problem. We just need
clean-up-fe-show to take those changes into account. And we now have
the choice of releasing
List.fold_until first, or of releasing them together,
by first releasing
List.fold_until, and then releasing
$ fe release jane/List.fold_until/clean-up-fe-show $ fe release jane/List.fold_until
Sometimes, it just doesn’t make sense to have one release process for your entire tree. For example, you might have a project that needs a certain amount of user-testing before you’re comfortable doing a release. That testing may uncover small changes you need to make along the way. But releasing those changes to the tip of the root feature may require you to pull in unrelated changes that would invalidate more of your testing.
With Iron, we handle this by minting a feature that corresponds to the release
process for the project in question. That way, you get full control over how
changes move into your feature. For developing Iron itself, releases are done
out of the
jane/fe feature. The owners of that feature decides which of the
jane/fe get released into it. And they can rebase
whenever they’re ready to pull in the most recent changes that have hit
After rebasing, they can also release the changes that were done within
jane, so others can benefit from them. All of this is under the
explicit control of the owners of
jane/fe, which allows them a lot of
independence, while still giving them tools to easily integrate their work with
Doing rebases and releases by hand can be a drag, especially for a feature like
jane that are released into quite often.
To deal with this, we have a continuous release process for
jane that is
automated by Hydra, our build-bot. When a feature is proposed for release, Hydra
does the following:
- Rebases the feature against its parent
- Checks that the resulting feature builds and all tests pass
fe release, which does its own checks as described earlier, including checking that the feature is fully reviewed.
If anything fails, the submitter is emailed with the details, and Hydra will try again if there are any changes either to the feature itself or to its parent.
Given that we do releases out of features other than
jane, there’s no reason
not to have support continuous releases in those places too. Accordingly, Iron
lets you mark any feature as a continuous release feature, which signals to
Hydra that it should run the continuous release process for the feature in
Why Iron matters
Iron is a pretty big improvement for us. Part of this is just that the system it replaces was old and crufty and had significant performance problems. But it’s more than that. Iron does a good job of breaking down the sometimes messy process of reviewing and releasing software into simple, easy to understand pieces. Given how fundamental these processes are to the act of developing software, I think that this is a real contribution.
The software itself is going to be open-sourced in the next month or two, but I’m hopeful that the ideas behind Iron will have some value as well. I’ve been struck in the last decade of working as a professional software engineer at how little is written about these issues. Outside of organizations that have built high quality tools to solve these problems internally, I think these questions are not that widely understood.