State of dhall-golang


#1

As you may be aware, I’ve been working on a new implementation of Dhall in Go. I thought it would be worth giving an update on where I’ve got to with it and where I see things going.

Summary

dhall-golang is a feature-complete implementation of the Dhall standard. It can parse, typecheck and evaluate Dhall expressions. However, the language bindings - ie unmarshalling Dhall expressions to Go data and vice versa - are still very much a work-in-progress.

Motivation

My own motivation for implementing dhall-golang is largely personal: I find Dhall a fascinating project, but if it is to be a YAML-killer, it needs to have language bindings in the primary language for most of the tools I use: Kubernetes, Prometheus, Concourse and Cloud Foundry are all written in Go. Therefore, Dhall bindings for Go seemed like an obvious thing to work on.

What currently works

Dhall-golang can parse, typecheck and normalize Dhall expressions. The expressions are parsed into the various data types in the ast package (though the parser itself is in the parser package).

Standard conformance

Dhall-golang is up to date with the standard v10.0.0.

The file spec_test.go runs the spec conformance tests. The expectedFailures variable lists the remaining spec tests that don’t pass, with comments as to why. In particular:

Marshalling and unmarshalling

unmarshalling.go has some basic support for unmarshalling Dhall values into your own Go data structures. However it is not as polished as I would like (more on this below).

Current active development

Normalization by Evaluation

I’m currently changing the core normalization algorithm to normalization-by-evaluation, much as dhall-haskell has already done. One difference is that I’m also planning on changing the typechecking algorithm too (currently, dhall-haskell uses NbE for normalization but still has the traditional typechecking algorithm).

Better unmarshalling

The unmarshalling support that exists is pretty basic, and I want to make this a lot better. In particular:

  • it doesn’t support go struct tags. Ideally I’d like to read the json: tags so that structs already tagged for unmarshalling from JSON can also be unmarshalled from Dhall. Additional dhall: tags may or may not be added too.

  • it’s in the main package which means it’s not actually importable from outside, oops. (Thankfully this is a quick fix)

  • I want to ensure that every Go type is convertible to and from Dhall so that, for example, you can pass a Go function in to a Dhall one and vice versa.

“Official” status milestone

As a result of the above, although I’m really happy with how dhall-golang is shaping up, I don’t think it’s quite ready for “official” status. In particular, I think better unmarshalling is a hard requirement for an official release. I also think it needs better documentation.

At that point, I can cut a tag and issue a PR to the official dhall-lang website to recognise dhall-golang as a supported Dhall implementation.

Possible future directions & places people could help

This is just a list of my random thoughts about where this could go in future. Some of them may be opportunities for people to contribute, although before NbE is finished the code will be subject to a huge rewrite which will make contributing difficult.

using

I have no need for this feature, and I am mildly concerned about it from a security point of view, so I do not plan to implement it myself. That said, if some has a need and wishes to contribute the feature, I would accept that PR.

Go style

This is my first major Go project so I’m probably not writing very idiomatic code. I would appreciate guidance on this from experienced gophers.

kubectl but dhall

There is already a dhall-kubernetes project. I think a possibly exciting way that dhall-golang could combine with it is by forking kubectl to read Dhall code natively, rather than having a workflow where everything has to compile to yaml first.


#2

Yeah, I also have reservations about the using keyword and holding off on implementing it would be a good opportunity to see if people actually need it in practice. Actually, even if people did request support for the using keyword I would still be open to other alternatives (like netrc files as I believe you previously mentioned).

I also agree that a Go binding is sorely needed, because I’m reasonably confident at this point that the quality of Dhall’s Kubernetes support will be instrumental in mainstreaming the language.


#3

Great to hear!

I think a possibly exciting way that dhall-golang could combine with it is by forking kubectl to read Dhall code natively, rather than having a workflow where everything has to compile to yaml first.

I’m interested in how you see this working. At work, we’ve come up with some interesting ideas about packaging objects as functions, so that crucial information can be supplied at installation (unknown at the time of packaging). Dhall makes this pretty simple, and it would be interesting to see how patterns like it could be exploited in a fork. (We’ll have something presentable hopefully within a few months).


#4

I think the first task is a fairly dumb port of kubectl that just uses the same Go structs and the same Dhall-kubernetes types (and it’ll Just Work since both are generated from the same Swagger spec :crossed_fingers:t2:).

In terms of passing functions around, I have some working proofs of concept for unmarshalling Dhall functions as Go functions. I also have some thoughts about marshalling Go functions to Dhall functions, though nothing working yet. I think anything that used these features would be more involved than a dumb port of kubectl, but I think it could very much be possible.