State of dhall-golang

#1

As you may be aware, I’ve been working on a new implementation of Dhall in Go. I thought it would be worth giving an update on where I’ve got to with it and where I see things going.

Summary

dhall-golang is a feature-complete implementation of the Dhall standard. It can parse, typecheck and evaluate Dhall expressions. However, the language bindings - ie unmarshalling Dhall expressions to Go data and vice versa - are still very much a work-in-progress.

Motivation

My own motivation for implementing dhall-golang is largely personal: I find Dhall a fascinating project, but if it is to be a YAML-killer, it needs to have language bindings in the primary language for most of the tools I use: Kubernetes, Prometheus, Concourse and Cloud Foundry are all written in Go. Therefore, Dhall bindings for Go seemed like an obvious thing to work on.

What currently works

Dhall-golang can parse, typecheck and normalize Dhall expressions. The expressions are parsed into the various data types in the ast package (though the parser itself is in the parser package).

Standard conformance

Dhall-golang is up to date with the standard v10.0.0.

The file spec_test.go runs the spec conformance tests. The expectedFailures variable lists the remaining spec tests that don’t pass, with comments as to why. In particular:

Marshalling and unmarshalling

unmarshalling.go has some basic support for unmarshalling Dhall values into your own Go data structures. However it is not as polished as I would like (more on this below).

Current active development

Normalization by Evaluation

I’m currently changing the core normalization algorithm to normalization-by-evaluation, much as dhall-haskell has already done. One difference is that I’m also planning on changing the typechecking algorithm too (currently, dhall-haskell uses NbE for normalization but still has the traditional typechecking algorithm).

Better unmarshalling

The unmarshalling support that exists is pretty basic, and I want to make this a lot better. In particular:

  • it doesn’t support go struct tags. Ideally I’d like to read the json: tags so that structs already tagged for unmarshalling from JSON can also be unmarshalled from Dhall. Additional dhall: tags may or may not be added too.

  • it’s in the main package which means it’s not actually importable from outside, oops. (Thankfully this is a quick fix)

  • I want to ensure that every Go type is convertible to and from Dhall so that, for example, you can pass a Go function in to a Dhall one and vice versa.

“Official” status milestone

As a result of the above, although I’m really happy with how dhall-golang is shaping up, I don’t think it’s quite ready for “official” status. In particular, I think better unmarshalling is a hard requirement for an official release. I also think it needs better documentation.

At that point, I can cut a tag and issue a PR to the official dhall-lang website to recognise dhall-golang as a supported Dhall implementation.

Possible future directions & places people could help

This is just a list of my random thoughts about where this could go in future. Some of them may be opportunities for people to contribute, although before NbE is finished the code will be subject to a huge rewrite which will make contributing difficult.

using

I have no need for this feature, and I am mildly concerned about it from a security point of view, so I do not plan to implement it myself. That said, if some has a need and wishes to contribute the feature, I would accept that PR.

Go style

This is my first major Go project so I’m probably not writing very idiomatic code. I would appreciate guidance on this from experienced gophers.

kubectl but dhall

There is already a dhall-kubernetes project. I think a possibly exciting way that dhall-golang could combine with it is by forking kubectl to read Dhall code natively, rather than having a workflow where everything has to compile to yaml first.

6 Likes
#2

Yeah, I also have reservations about the using keyword and holding off on implementing it would be a good opportunity to see if people actually need it in practice. Actually, even if people did request support for the using keyword I would still be open to other alternatives (like netrc files as I believe you previously mentioned).

I also agree that a Go binding is sorely needed, because I’m reasonably confident at this point that the quality of Dhall’s Kubernetes support will be instrumental in mainstreaming the language.

#3

Great to hear!

I think a possibly exciting way that dhall-golang could combine with it is by forking kubectl to read Dhall code natively, rather than having a workflow where everything has to compile to yaml first.

I’m interested in how you see this working. At work, we’ve come up with some interesting ideas about packaging objects as functions, so that crucial information can be supplied at installation (unknown at the time of packaging). Dhall makes this pretty simple, and it would be interesting to see how patterns like it could be exploited in a fork. (We’ll have something presentable hopefully within a few months).

#4

I think the first task is a fairly dumb port of kubectl that just uses the same Go structs and the same Dhall-kubernetes types (and it’ll Just Work since both are generated from the same Swagger spec :crossed_fingers:t2:).

In terms of passing functions around, I have some working proofs of concept for unmarshalling Dhall functions as Go functions. I also have some thoughts about marshalling Go functions to Dhall functions, though nothing working yet. I think anything that used these features would be more involved than a dumb port of kubectl, but I think it could very much be possible.

3 Likes
#5

hi @philandstuff, can I ask what the status of this is?

We have some work in Go so I tried to take a look at the work you’ve done here already.

In particular, the lack of go struct tags is a deal-breaker, since it makes it impossible to unmarshall Dhall with fields starting with lower-case letters since the corresponding field in the struct isn’t exported. I guess the way forward for me at the time being is to write out a Config struct, using JSON struct tags, and script a dhall-to-json conversion outside of the Go code for the time being. Obviously this isn’t ideal.

If you don’t have time to work on go struct tag support, do you figure it to be difficult / time-consuming to add? I’m wondering if I could get support at work to spend some time to implement it, though I wouldn’t get more than a few days. The idea being to justify the time spent in order to prevent needing to write dhall -> json -> go glue code.

For the time being, if json struct tags are sufficient, then I’m not too concerned about unmarshalling functions at this time.

#6

Hi and thanks for the interest!

I’ve been going pretty slowly with dhall-golang recently. The NbE refactor is complete and merged into master and other than that I’ve been trying to keep up with the new language versions, and improving the godoc, which has also prompted questions about package and API structure (eg sometimes it’s easier not to expose something in the first place rather than to document it).

As a result I haven’t spent much time looking at unmarshalling. I know the current unmarshal.go needs an overhaul, possibly a complete rewrite, which is work I want to do before getting to struct tags.

Also, in my personal life I’m expecting a second child this Spring, during which I’ll be taking shared parental leave to look after the baby. This may mean either more or less time to work on dhall-golang, it’s hard to predict, but it does make me hesitant to commit to anything. I’d like to get better unmarshalling in the first half of this year, but I’m making no promises :slight_smile:

3 Likes
#7

So I got some time to look at struct tags and they turned out to be not as hard as anticipated. Take a look at the dhall-golang godoc, particularly the “Example (Tagged)”.

I think this is a real milestone, and I think that with struct tags dhall-golang is pretty close to a v1.0 release. If you think you might find dhall-golang useful, please give it a go and give me feedback in this thread so I can do quality-of-life tweaks before cutting a proper release.

4 Likes
#8

I’ve added a line to my netrc, but this hasn’t helped me to use dhall-golang via the terraform dhall provider. The lack of using also means my dhall files can’t be read by dhall-haskell since Network.HTTP.Client doesn’t seem to have any documentation on supporting netrc files.

I was excited by the idea of using the netrc, since it would enable a more flexible approach to dependencies in Dhall.

What is the latest guidance on this for dhall-golang and dhall-haskell?

#9

I just saw https://github.com/dhall-lang/dhall-haskell/pull/2236, so that answers half of my question.