Announcing dhall-render: a more powerful `dhall to-directory-tree`

#1

For a while (almost immediately after it was released) I’ve been using dhall to-directory-tree for the purposes of abstracting away CI boilerplate (travis, github-actions, Dockerfile, cloudbuild, etc).
You know, stuff that’s usually copy/pasted around and modified, isn’t always trivial to get right, and is generally a pain to maintain when you have more than a handful of repositories that are all more-or-less built in the same way.

I recently went about trying to merge my own WIP github-actions schemas with https://github.com/regadas/github-actions-dhall, and hit a snag: to use dhall-to-directory-tree, you can’t just output records, you need to output JSON.Type. Which means you need a flurry of *toJSON functions taking in your dhall types and generating JSON. And those can cause some unfortunate performance issues.

As I was adjusting my existing toJSON functions to fit the different schema I decided maybe it’s just not worth the hassle at this point just so I can use dhall to-directory-tree - it’d be easier for me to replace that, and lean on dhall-to-json for automating the JSON creation.

So I wrote a ruby wrapper to basically take the output of dhall-to-json for a simple schema, and do what dhall-to-directory-tree does but with some extra bells and whistles (like installing symlinks to the generated files, and allowing executable files).

So here it is! dhall-render. It’s still a work in progress, but I’m intending to adopt it to replace my own use of dhall to-directory-tree.

If the name dhall-render is a bit too canonical-sounding for such a minor project I’m happy to take suggestions. Ultimately I’d quite like to see these features integrated into dhall to-directory-tree itself, though maybe some of them are a bit too opinionated (particularly the symlink generation).

8 Likes
#2

Looks neat!

For context:

  • I didn’t use travis for a while and never looked at github-actions, yet
  • Where I work, we use gitlab CI and I solved the same problem with a shared toolbox git submodule. It’s a bit of a mess, sometimes. I’d like to try something like dhall-render.
  • Gitlab-ci resolve submodules first, then parses .gitlab-ci.yml which can include fragments from the 'toolbox`
  • the CI jobs rely mainly on a Makefile that also include fragments from the toolbox and calls shell scripts defined in the toolbox.
  • this works only because of the resolution order (hard-coded in gitlab-ci) : submodules first then .gitlab-ci.yml
  • we update the toolbox manually in each project(we check it’s up-to-date on tags only)

My question is: when do you run dhall-render ? In a pre-commit hook? (this boils down to this question).

#3

Oh right, yeah I guess that wasn’t clear. A pre-commit hook would work, but I dislike git hooks. Whenever you change the dhall expression, you need to re-run the script / make task. Plus CI will always regenerate, and fails if there’s an uncommitted diff.

This does mean you need to synchronize your dhall versions between users and CI (otherwise dhall format might legitimately produce different results), which is easy enough if you are using docker for both.

#4

I think I’d enforce this with an early check in the CI pipeline that fails if the directory tree is not up-to-date with the Dhall expression. For this to work we need to know the state of the dhall expresson when the tree was last generated.

Maybe dhall-render could optionaly dump the sha256 hash of the expression in generated/.dhall-render.sha256 ?

The downside of this approach is that the checking job would require to be able to resolve http imports, which is sometimes hard in the context of a CI job.

#5

You mean the actual file digest right, not the dhall expression? (since you imply you want to do this without resolving imports). That would only work for single-file expressions, i.e. no local imports.

I’ve found that simply re-running the generation and checking for a git diff is sufficient, and catches other issues (people accidentally modifying generated code). You’re right about CI needing to resolve imports, if that’s a problem you may have to implement your own file-digest based checks.

1 Like
#6

@timbertson: Very nice! I can advertise dhall-render from the dhall_lang Twitter account if you don’t mind

1 Like
#7

Sure, thanks :grinning:

2 Likes
#8

Thanks! I will have a look at this, I’ve been wanting something like this for a long time :slight_smile:

(Also good to see people using Dhall for CI and github actions in particular!)

#9

BTW, I’ve just stumbled across another tool that seems similar to dhall to-directory-tree, but using Nix instead of Dhall: http://hackage.haskell.org/package/nix-freeze-tree

#10

@sjakobi from the readme it sounds almost like the opposite – I have some files, create a nix expression to reproduce them. Like a directory-tree-to-nix