Rules_dhall - bazel/dhall integration

muff1nman · December 22, 2020, 7:23am

Hi Folks!

I have been improving on Jasper Humphrey’s initial dhall integration with bazel and have made quite a few improvements in a fork here: https://github.com/muff1nman/rules_dhall. I think its gotten to a mostly usable state (or at least its not entirely broken) and wanted to share.

Why bazel?

There’s a couple of advantages of having dhall integrate with a generic build system like bazel:

Allows tighter integration into build pipelines with other non-dhall code and allows for glueing dhall with other build processes.
Can utilize bazels caching/dependency mechanism. While dhall has caching, using something like bazel allows you to fetch a zip/tar of a repository (i.e dhall-kubernetes) in a fraction of the time that it takes for dhall to traverse github’s raw links while getting rate limited.
Related to the above, it allows a more declarative way of managing dependencies. Instead of having a bunch of ../ path traversals, one can utilize bazels target definitions to refer to dhall files in a more root-centric and portable manner (or even across repositories without having to use http within dhall).

Of course with that said there are a couple of rough edges.

In order to get freeze to work I have made use of env: imports that “refer” to bazel dependencies. This makes running dhall without bazel a bit unwieldy. This is worsened by the fact that bazel maintains a separate dhall cache which means your hashed imports don’t resolve. However the one benefit of going this route is that I can “freeze” an entire repository in one script while following the dependency chain - see tools/freeze-all.sh. Note that this is technically optional (for now) as if you don’t use the bazel freeze targets, things should work fine as they did in the initial version of rules_dhall.
I’m still working on figuring out a good way to make dhall-docs output a more holistic view of all your dhall “libraries”
and probably quite a few others that are probably more specific to kubernetes

At any rate, let me know if you have any questions. I’ll also take any suggestions

Gabriel439 · January 8, 2021, 4:32am

@muff1nman: The main suggestion I would make is to note that there is a way to do this that is compatible with open source Dhall code. You can find the long explanation here:

…and you can also compare that against the implementation here:

github.com

NixOS/nixpkgs/blob/master/pkgs/development/interpreters/dhall/build-dhall-package.nix

{ dhall, dhall-docs, haskell, lib, lndir, runCommand, writeText }:

{ name

  # Expressions to add to the cache before interpreting the code
, dependencies ? []

  # A Dhall expression
  #
  # Carefully note that the following expression must be devoid of uncached HTTP
  # imports.  This is because the expression will be evaluated using an
  # interpreter with HTTP support disabled, so all HTTP imports have to be
  # protected by an integrity check that can be satisfied via cached
  # dependencies.
  #
  # You can add a dependency to the cache using the preceding `dependencies`
  # option
, code

  # `buildDhallPackage` can include both a "source distribution" in

This file has been truncated. show original

Also, you might find this thread useful, which was the origin of that work:

The trick that the Nixpkgs integration uses to build open source Dhall code without resolving HTTP imports is to add all dependencies to the cache and enforce that all HTTP imports are satisfied via the cache. Specifically, what it does is:

Require all HTTP imports to have integrity checks

Nixpkgs builds dhall packages with HTTP imports disabled (i.e. using a special build of the Haskell dhall package configured with -f-with-http)
Require the user to specify all dependencies (remote or otherwise) explicitly using the build system (e.g. Nix or Bazel)

In other words, the dependencies are not inferred from the Dhall code (although the dhall-haskell project provides a dhall-to-nixpkgs utility for automating the task of generating the Nix dependencies from the Dhall code).
Ensure that all dependencies are added to a build-local cache when building each Dhall package
If an HTTP import is resolved, it’s either fetched from the cache or the build fails

… and that’s basically it! This trick works with all open source Dhall packages and is also dhall freeze-compatible. It also plays nice with build systems like Nix and Bazel because you can use the build system to fetch repositories and also let the build system take care of threading and caching dependencies.