Reviving dhall-concourse

We have revived our repository that maintains types for Concourse pipelines back into active maintenance status: https://github.com/coralogix/dhall-concourse

A couple years ago, when we first attempted to write this, we ended up in a place (v0.3.0) where the performance of the implementation was atrocious and basically un-runnable in practice, so we worked around the issue by creating ad-hoc types for pipelines as needed. The main reason why was because attempting to encapsulate all of the possible types for external resources produced normal forms where the unions were seven levels deep (i.e. < A : < B : < C : < D : < E : < F : < G : Text > > > > > > >) and the normal forms were far too large. And our solution back then didn’t deal with the recursive nature of Concourse’s steps very well.

The main thing that has changed between then and now is a) support for schemas b) freezing c) my brain finally clicking on how to deal with the recursive type issue in a satisfactory way. As a result, we’ve cut the dhall-to-yaml runtime down to under 2.5 seconds :tada: which makes it more than satisfactory for production purposes.

The interesting stuff we use dhall-concourse for:

  • Showing that the jobs for building a master branch and a PR are exactly the same, except for updating PR status
  • Conditionally adding production deployments only for config that declares itself to be ready to deployed to production
  • Extracting (from parameter configuration) and intertwining different Slack channels, to ensure that notifications are sent to the correct parties (i.e. service owners are usually sent notifications, slack channels corresponding to environments are sent notifications for deployments to those environments, Concourse admins are sent notifications whenever Concourse errors a job for internal reasons), something which is so difficult and annoying to maintain that Concourse’s maintainers are on a multi-year design trip (see: https://github.com/concourse/rfcs/discussions/10 and https://github.com/concourse/rfcs/pull/37 in part) to try and make ergonomic, but which Dhall helps make much, much easier for us to maintain consistently.
  • Combined with the Github List Repos Concourse Resource to automatically generate pipelines for repositories as they are created

Hope it’s useful to other people in the community as well.

5 Likes

Sounds very relevant to my interests! Are your uses of it in public repos somewhere we can look at?

Sorry, I still don’t have approval to open-source the pipelines themselves. I can try to write a few more simple examples, though, if that would be helpful.

2 Likes

I wrote up some simple pipelines here: https://github.com/coralogix/dhall-concourse/tree/master/examples , which generate the examples in the Concourse documentation.

2 Likes

I have a pull request up to mention dhall-concourse on dhall-lang.org: https://github.com/dhall-lang/dhall-lang/pull/1042

1 Like