I’m using Dhall in a corporate environment, and we’re finding that the compilation performance for large Dhall codebases has become a bit of a problem. Two of our Dhall repos have compile times of at least 10 minutes each, and will sometimes error out during building with a 137 (out of memory) return code. I’ve already tried to break up the code into separate, more manageable-sized packages as best I can. I understand that there’s likely to be no silver bullet for this, but these two repos have not reached their logical maximum size and I’m concerned that this problem may get exponentially worse as we add more code.
One contributing factor to slow compile times that has come up repeatedly in the past is very large normal forms being serialized to the cache (e.g. ~/.cache/dhall), especially for projects that use dhall-kubernetes (since it has large normal forms, especially for certain imports). If that were the case then something like this would alleviate the problem:
… because it would allow opting out of storing normal forms so the cache products would be quicker to deserialize.
Does that sound similar to the issue that you’re having or do you suspect that it is something else?
Hello - thanks for the quick response - I’m building the dhall packages via nix (with buildDhallPackage, which I think already alpha-normalizes the source dhall expression before converting it to a binary: nixpkgs/build-dhall-package.nix at master · NixOS/nixpkgs · GitHub
so I’m not actually sure that this feature would make a difference.
I still think the feature would make a difference because implementing that feature would also entail extending the cache to support expressions with unresolved import references (so long as they’re protected by an integrity check) (i.e. the sha256:none:… checks mentioned in that issue). Then you could extend the Nixpkgs support for Dhall to use those unresolved cache entries and sha256:none integrity checks to benefit from the performance improvement. However, all of this would be a lot of work which has not been standardized or accepted, so keep that in mind when setting expectations.