As with many projects in the OCaml world, at Jane Street we have been working on migrating from camlp4 to ppx. After having developed equivalent ppx rewriters for our camlp4 syntax extensions, the last step is to actually translate the code source of all our libraries and applications from the camlp4 syntax to the standard OCaml syntax with extension points and attributes.
For instance to translate code using
pa_test, we have to
TEST = <:test_result< int >> ~expect:42 (f x)
let%test _ = [%test_result: int] ~expect:42 (f x)
For small to medium projects it is enough to just take a couple hours to translate the source code by hand. But at Jane Street where we have a huge OCaml code base making extensive use of camlp4, it is simply not realistic. So we needed a tool to do that for us.
Writing a tool to automatically convert the syntax
Since the output of such as tool has to be accepted as the new code source that is committed in our repository, it must preserve the layout of the original file as much as possible and of course keep the comments. This mean that any approach using an AST pretty-printer would be extremely complex.
The path we choosed is to textually substitute the foreign syntaxes in the original file for the new ones. One could imagine doing that with a tool such as sed, awk, perl, … however doing it properly would be fastidious and it would be pretty-much impossible to be 100% sure it would never translate things it is not supposed to. Plus writing perl is not as fun as writing OCaml programs.
Instead there is an easy way to find the foreign syntaxes: using camlp4 itself. To subsitute the text of foreign syntaxes the only thing we need to know is their location in the original file, and camlp4 can help us with that.
Writing dummy camlp4 syntax extensions
The idea is to write for each camlp4 syntax extension a dummy one that define the same grammar productions as the real one, but instead of generating code it simply record substitutions at certain locations.
Then we do the following:
- parse a file with camlp4 and our dummy syntax extensions
- apply all the recorded substitutions to the original file
This approach has the advantage of interpreting the original file in the exact same way as our regular syntax extensions. Giving us good confidence we did not change the syntactic constructions by mistake.
To do so we define this API:
(** [replace loc repl] records a text substitution that replaces the
portion of text pointed by [loc] by [repl]. *)
val replace : Loc.t -> string -> unit
Then writing a dummy camlp4 syntax extension is pretty easy. For instance for a
test: [[ "TEST" -> replace _loc "let%test" ]];
[[ `STRING _; "=" -> ()
| "=" -> replace _loc "_ ="
[[ test; name_equal; expr -> <:str_item< >>
On the fly conversion and diffing the generated code
Since this tool was convenient to use, we used it to check that our newly written ppx rewriters did the same thing as the old camlp4 syntax extensions:
- for a given OCaml source file of a library or application, we converted it
camlp4-to-ppxand saved the result
- we processed the original file using camlp4 and the translated one using our ppx rewriters
- in both case we saved the output of
-dparsetree(human-readable version of the internal OCaml AST) and
- we diffed the camlp4 and ppx outputs of
-dparsetree, as well as the outputs of
This was all quite easy to do with jenga. We kept looking at the generated diffs until they were all empty.
We have quite a lot of code in our camlp4 syntax extensions and converting them to ppx was a long mechanical job and so quite error-prone. Given that this diffing turned out to be really helpful to find errors.
The Camlp4 Syntax is not quite the OCaml syntax
While using this we noticed that quite a few syntaxes accepted by camlp4 are not accepted by OCaml, for instance:
let _ x = x
let f l = List.map l ~f:fun x -> x + 1
These where quite easy to fix automatically as well using
Github repo and extension
We published a slightly modified version of this tool on github.
The method we used doesn’t work out of the box with all syntax extensions. For
instance to convert code using
lwt.syntax some more work needs to be done on
camlp4-to-ppx. But it is a good starting point.