Help with `buildDhallPackage`

[yuuki]

I’m trying to generate a YAML file from a Dhall expression in Nix. The Dhall file that I pass to dhall-to-yaml is shown below and I will refer to it as “dhall source”

let Alacritty =
      https://raw.githubusercontent.com/cideM/dhall-alacritty/master/linux.dhall sha256:c9cf010f9ef1a7da4590d70571c1fa5114743f9096b8387b180bfb02f5cdffb1

let mono = ./mono.dhall with size = 12.0

in  Alacritty.Config::{
    , font = mono
    , shell = Some { program = "/usr/bin/fish", args = [ "-l" ] }
    , colors = ./papercolor.dhall
    , key_bindings = ./keys_common.dhall
    }
  with window.decorations = Alacritty.Window.Decoration.full
  with window.dynamic_padding = True
  with window.padding = { x = +10, y = +10 }

I made sure that everything in the dhall-alacritty dependency has a sha256 for the Dhall Prelude, so that I can use buildDhallPackage to load these things into the cache.

Here’s the Nix code

let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs {};

  prelude = pkgs.dhallPackages.buildDhallPackage {
    name = "dhall-lang-prelude";
    code = "${sources.dhall-lang}/Prelude/package.dhall";
  };

  # Fix in source
  preludeMap = pkgs.dhallPackages.buildDhallPackage {
    name = "dhall-lang-prelude";
    code = "${sources.dhall-lang}/Prelude/Map/Type";
  };


  linux = pkgs.dhallPackages.buildDhallPackage {
    name = "dhall-alacritty";
    code = "${sources.dhall-alacritty}/linux.dhall";
    dependencies = [
      prelude
      preludeMap
    ];
  };

  # config = pkgs.dhallPackages.buildDhallPackage {
  #   name = "alacritty_linux";
  #   code = ./src/linux.dhall;
  #   dependencies = [ 
  #     linux
  #   ];
  # };

in derivation {
  name = "alacritty_linux.yml";
  builder = "${pkgs.bash}/bin/bash";
  args = [ ./builder.sh ];
  dhalljson = pkgs.haskellPackages.dhall-json;
  coreutils = pkgs.coreutils;
  inherit linux;
  # inherit config;
  src = ./src;
  system = builtins.currentSystem;
}

The idea is to use that derivation in Home Manager to generate an alacritty.yml file. The problem is that if I uncomment alacritty_linux it complains about missing ./mono.dhall. So in other words, the dhall source file expects other files to be available in the same folder. If I run this file through buildDhallPackage it ends up in the Nix store, without the rest of the sources. But if I comment out the call to buildDhallPackage, my cache doesn’t have the necessary dependencies, it tries to make an HTTP call and that fails for obvious reasons in the Nix builder.

I’m missing a piece of the puzzle here and I don’t know what it is. My idea is that I should uncomment the commented out lines, and then pass inherit config to derivation but while also adding the necessary, local (~ relative) sources to name = "alacritty_linux"

[Gabriel439]

@yuuki: What’s happening is that Nix has special language support for path literals that interferes with what you are trying to do.

Specifically, when you write something of the form:

  config = pkgs.dhallPackages.buildDhallPackage {
    name = "alacritty_linux";
    code = ./src/linux.dhall;
    dependencies = [ 
      linux
    ];
  };

… Nix will first add ./src/linux.dhall to the Nix store in a process that is essentially the same as running this command:

$ nix-store --add ./src/linux.dhall
/nix/store/…-linux.dhall

… and then it will replace the path literal with the path that it added to the Nix store, like this:

  config = pkgs.dhallPackages.buildDhallPackage {
    name = "alacritty_linux";
    code = "/nix/store/…-linux.dhall";
    dependencies = [ 
      linux
    ];
  };

… and that will then fail to resolve the ./mono.dhall relative import because ./mono.dhall was not also added to the Nix store. Also, even if mono.dhall had been added to the /nix/store the relative import would fail because it’s going to be looking for /nix/store/mono.dhall when it resolves that relative import and not /nix/store/…-mono.dhall.

Fortunately, there are two ways you can work around this.

The first solution, is to add the entire ./src directory to the Nix store instead of the file, like this:

  config = pkgs.dhallPackages.buildDhallPackage {
    name = "alacritty_linux";
    code = "${./src}/linux.dhall";
    dependencies = [ 
      linux
    ];
  };

That will create a /nix/store/…-src directory containing both linux.dhall and mono.dhall underneath it, so that relative references work correctly. I’m guessing that this is probably the solution that you want.

However, for completeness, the second solution is that you can protect ./mono.dhall with an integrity check and then add a package that builds ./mono.dhall as a dependency of the package that builds linux.dhall. In other words, the cache resolution trick that was created for remote imports also works for local imports. For example, we do this at work because we have a few Dhall packages in disparate parts of our monorepo, and rather than add the entire repository to the /nix/store we just protect “distant” imports with integrity checks so that they can be resolved via Nix dependencies.

[yuuki]

You are the most helpful person on the entire internet. Thanks for the very well written reply. I was ultimately able to make things work thanks to your help. I also needed to change my builder.sh from

"$dhalljson/bin/dhall-to-yaml" --file $src/linux.dhall --output $out

to

"$dhalljson/bin/dhall-to-yaml" --file $config/source.dhall --output $out

so it refers to the built Dhall package source. For that, I also added the source = true; parameter. The difference is that $src/linux.dhall would be a source file starting with let Alacritty = https://raw.githubusercontent.com/cideM/dhall-alacritty/master/linux.dhall sha256:c9cf010f9ef1a7da4590d70571c1fa5114743f9096b8387b180bfb02f5cdffb1. My assumption was that this would work, since the import should be cached. But the call to dhall-to-yaml seems to still try to fetch this via remote.

I then concluded that I need to use the buildDhallPackage infrastructure to prebuild everything, so that the final source passed to dhall-to-yaml in the builder has no remote calls. And indeed, if I add a simple cat $config/source.dhall inside the builder, I get something like

{ colors =
  { bright =
    { black = "0x949494"
    , blue = "0x4271ae"
    , cyan = "0x3e999f"
    , green = "0x718c00"
    , magenta = "0x8959a8"
    , red = "0xd7005f"
    , white = "0xf5f5f5"
    , yellow = "0xd75f00"
    }
...

Which is the output you’d get if every import and every function call is evaluated. The only step missing here is to actually serialize this to YAML, which is then done by dhall-to-yaml. I checked and what lands in the Nix store is indeed my alacritty.yml.

I’ll update this post with a link to the whole folder once I’ve cleaned some stuff up and pushed it.

Thanks again for the help!

Here’s a permalink to the final Nix file: https://github.com/cideM/dotfiles/blob/79eff97b97d7278ef3e5d4b5bd3b36011dd2f94f/src/nixpkgs/.config/nixpkgs/alacritty/default.nix#L1

The folder structure here is super messy but the concept works.

[Gabriel439]

You’re welcome!

[Gabriel439]

If anybody stumbles across this, there is now official Nixpkgs support for Dhall which takes care of all of this for you: https://nixos.org/manual/nixpkgs/unstable/#sec-language-dhall

[Chris_Harris]

Hello - one thing which I don’t think is addressed here or in the Nix documentation, but which is hinted at in your response above, is how to import a dhall package that exists in the nix store into another dhall file. I’ve been playing around with this and have tried the following, so far without success:

• created a single test.dhall file
• created a flake that defines a test_dhall_packaged in Nix, via buildDhallPackage on test.dhall
• run nix build to get test_dhall_packaged into the Nix store
• … and this is where things start to go less smoothly…
• created a new dhall file caller.dhall, in a new folder, that tries (incredibly optimistically) to import test.dhall via a simple test_dhall = ./test.dhall. This doesn’t work, obviously
• the documentation implies that Nix somehow looks up the imports based on the hash, so I added a hash to the above import, and it’s now test_dhall = ./test.dhall sha256:122057f.... One problem with this was that I didn’t know where to actually get the hash from. There was a hash-like filename in the ./result/.cache/dhall directory of my test_dhall_packaged folder, so I used that.
• since this didn’t work, I then went a step further and created a flake to package up test.dhall as well, and added test_dhall_packaged in the dependencies parameter of its buildDhallPackage. I was optimistic about this but ultimately when I ran nix build it worked about as well as two steps above, ie, not at all.

I assume that what I’m ultimately trying to do here (import a dhall package defined in the nix store into another dhall package) is possible, and that there’s some more ergonomic way to do it than by importing directly from the nix store - could you point me in the right direction as to how to do it?

[Gabriel439]

@Chris_Harris: The simplest solution is to build a directory instead of one file at a time. The dhall-to-nixpkgs tool has a directory subcommand for this purpose, and when you build an entire directory (e.g. one containing your two files), the Dhall files can refer to each other using relative imports without issues.

There is a way to do what you want, though, which is that you can build one Nix package per Dhall file and one Dhall file packaged by Nix can depend on another Dhall file packaged by Nix, so long as the dependency is protected by an accurate integrity check.

I’d need more details (i.e. a sample reproduction of the failure) to give more accurate feedback, but it sound to me like the thing you were missing is to add the test.dhall Nix derivation as a dependency of the caller.dhall Nix derivation. Specifically, buildDhallPackage has a dependencies argument that you need to use if your Dhall file depends on any other Dhall files. Otherwise it will not know how to locate those dependencies and add them to the cache to resolve the integrity checks.

[Chris_Harris]

Thanks, the dhall-to-nixpkgs tool looks to be exactly what I need, not sure how I overlooked it. I’ll give it a go.

[Chris_Harris]

This works great, thank you. I wanted to make my dhall package available in the nix shell, so I’ve created a flake which looks a bit like this:

  inputs = {
    nixpkgs.url = github:NixOS/nixpkgs/nixos-22.05;
    flake-utils.url = github:numtide/flake-utils;
    dependency_dhall_package.url = https://repo.com/dependency_dhall_package;
  };
  outputs = { self, nixpkgs, flake-utils, dependency_dhall_package }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};

        pkg = pkgs.dhallPackages.buildDhallDirectoryPackage {
          name = "examples_dhall_package";
          src = ./.;
          file = "main.dhall";
          source = false;
          document = false;
          dependencies = [
            (pkgs.dhallPackages.Prelude.overridePackage { file = "package.dhall"; })
            dependency_dhall_package.packages.${system}.default
          ];
        };

      in {
        devShell = pkgs.mkShell {
          propagatedBuildInputs = [ pkg ];
          shellHook = ''
            cp ${pkg}/.cache/dhall/* $HOME/.cache/dhall/
          '';
        };
      });

Meaning that from within the shell, I can run eg
dhall <<< "(./main.dhall sha256:xyz...).some_attribute"
on the command line.

This all works fine, but is there a more canonical way of making the dhall package available in the devShell? Running cp from within the shellHook feels a touch hacky.

[Gabriel439]

There is not an ergonomic way to add the path built by the package to the cache search path. See: https://github.com/dhall-lang/dhall-lang/issues/1239

[Chris_Harris]

I am now at the stage of trying to manage multiple Dhall projects that are packaged via Nix. Could I check that my understanding is correct on one thing? If I have dhall package A which has package B as a dependency, package B obviously goes in the dependencies list in package A’s buildDhallPackage. Nix will build package B if need be, and then buildDhallPackage symlinks the dhall binary for package B from the nix store, to the user’s ~/.cache/dhall. So far so good.

However, package A’s source will still have package B’s hash hard-coded into it, ie, in package_a.dhall, there will be a line like let packageB = ./package_b.dhall sha256:1220123456789...

If I understand correctly, it’s the user’s job to make sure that this hard-coded hash value is kept up-to-date. If this value is not the hash of the version of package B that gets built during package A’s nix build process, then package A’s build may fail. Also, doing eg nix flake update on package A will appear to update package B, but actually the old version of package B will still be being used, at least until this hard-coded hash value is updated.

If so, this feels a little tricky. I’d have thought that it would be better for a nix update of package A’s dependencies to automatically update the version of package B being imported in package A, either by automatically updating package B’s hard-coded hash in package_a.dhall, or by Dhall not requiring a hard-coded hash in the first place when using Nix builds (somehow).

However I’m sure you’ve thought about this more than I have and I’d be interested to hear your view on whether this is something that should be fixed, or whether the current situation is the optimal one, all things considered.

[Gabriel439]

Your understanding is correct and there isn’t an automated way to keep the hash in the source code in sync with the hash of the Nix-supplied dependency. This is a limitation of the Nixpkgs support for Dhall

[Chris_Harris]

OK, thank you, that’s helpful to know.