Have we solved the problem of generated config yet? (best practices)

#1

Dhall promises to replace all kinds of configuration formats by one language.
However, in practice I need predefined formats to be checked into my source code repository.

For example, the CircleCI service expects a yaml file at .circleci/config.yml. All other CI systems I know work similarly.

Are there any best practices the dhall community recommends for these use-cases?

Questions:

  • When to generate this file? pre-commit hook?
  • How to set up CI so that the files don’t go out of sync? With a good workflow?
  • How to make it so that developers don’t have to think about this?
  • How to make developers not hate the fact that they have to check in generated code?
4 Likes
Announcing dhall-render: a more powerful `dhall to-directory-tree`
#2

Fwiw, we deal with this by using Concourse, which doesn’t follow a .concourse/config.yml kind of model. And I’m convinced that pipeline-stored-in-repo is a fundamentally poor model, because either a) improvements in one repo don’t carry over into other repos, b) instituting changes across all repos in a company quickly turns into a slow political death march, or c) developers are developers, not DBAs, not security professionals, etc.

To answer your question, I don’t think it’s solvable. Git client-side commit hooks cannot guarantee that the dhall executable is installed and available on the path, let alone of the correct version. GitHub doesn’t allow (not really) for server-side commit hooks (in a previous life I got some experience with BitBucket ScriptRunner hooks and everything else pales by comparison…). The best you can do is have a CI process which references itself, which is to say, has a pre-build validation step which checks a diff between the committed file and what would be generated, and a clear error message describing how to re-generate the file.

#3

@ari-becker some CI systems, at least Zuul, let you use git based configuration across multiple repos. Though I wish more services would accept dhall format directly, that would make our life easier :slight_smile:

@Profpatsch to check in generated code, e.g. go vendors copy, we have a CI jobs that look for clean git status after running the generation command. We use a simple Makefile target that is easy to run by the developer or the CI. The test works both way, it also fails if the yaml has been updated without using dhall.

#4

I use a Makefile in github-actions-dhall (I’m not sure if you are already familiar with it): https://github.com/vmchale/github-actions-dhall

Checking in generated code isn’t ideal but the only alternative I see is setting up some bot to go in-between, which, based on my experience with these things, would not be worth it.

1 Like
#5

@Profpatsch: You can study how we do this for the official Dhall language test suite. We have some .diag files which are generated from .dhallb (CBOR) files and they need to stay in sync.

First, we author a Nix derivation that builds the generated files, like this one:

Then, for user convenience we add a script to build that derivation and copy the generated files to where they belong:

Optional: make that script part of a git commit hook so that users are less likely to forget to do it.

Finally, we add a Nix derivation to CI that checks that the files built by the derivation match the files that should be generated:

That enforces that the generated files can’t get out of sync with the original files in case people forget to update the generated files.

#6

There is now a chapter in the Dhall manual addressing the original request:

2 Likes