Slow compile times

Hello all,

I’m using Dhall in a corporate environment, and we’re finding that the compilation performance for large Dhall codebases has become a bit of a problem. Two of our Dhall repos have compile times of at least 10 minutes each, and will sometimes error out during building with a 137 (out of memory) return code. I’ve already tried to break up the code into separate, more manageable-sized packages as best I can. I understand that there’s likely to be no silver bullet for this, but these two repos have not reached their logical maximum size and I’m concerned that this problem may get exponentially worse as we add more code.


One contributing factor to slow compile times that has come up repeatedly in the past is very large normal forms being serialized to the cache (e.g. ~/.cache/dhall), especially for projects that use dhall-kubernetes (since it has large normal forms, especially for certain imports). If that were the case then something like this would alleviate the problem:

… because it would allow opting out of storing normal forms so the cache products would be quicker to deserialize.

Does that sound similar to the issue that you’re having or do you suspect that it is something else?

Hello - thanks for the quick response - I’m building the dhall packages via nix (with buildDhallPackage, which I think already alpha-normalizes the source dhall expression before converting it to a binary: nixpkgs/build-dhall-package.nix at master · NixOS/nixpkgs · GitHub
so I’m not actually sure that this feature would make a difference.

Oh yeah, in that case that’s trickier to fix.

I still think the feature would make a difference because implementing that feature would also entail extending the cache to support expressions with unresolved import references (so long as they’re protected by an integrity check) (i.e. the sha256:none:… checks mentioned in that issue). Then you could extend the Nixpkgs support for Dhall to use those unresolved cache entries and sha256:none integrity checks to benefit from the performance improvement. However, all of this would be a lot of work which has not been standardized or accepted, so keep that in mind when setting expectations.

1 Like

Hello - we’ve now reached the point at which our config fails to build (with an out-of-memory return code) on our nix build server, which has 64GB of memory. The parts of the config which seem to be causing trouble are not the raw data, but rather some of the functions which operate on the data (eg, to update a given element in a list). The config is still fine and works fast at evaluation-time.

I don’t mean to sound like I’m pressuring you to fix this situation, and I’d imagine there are no easy solutions, but we are trying to make a decision on what configuration system to use for the next version of the company’s product, and this problem makes it hard for me to push for Dhall. I find this unfortunate because Dhall fits our use case in many other ways and it’s a pleasant and well-thought-through language to use. So I’m not sure if there’s any useful outcome to me saying this, but I thought you might want to receive this feedback.