Slow compile times

Chris_Harris · April 12, 2023, 2:29am

Hello all,

I’m using Dhall in a corporate environment, and we’re finding that the compilation performance for large Dhall codebases has become a bit of a problem. Two of our Dhall repos have compile times of at least 10 minutes each, and will sometimes error out during building with a 137 (out of memory) return code. I’ve already tried to break up the code into separate, more manageable-sized packages as best I can. I understand that there’s likely to be no silver bullet for this, but these two repos have not reached their logical maximum size and I’m concerned that this problem may get exponentially worse as we add more code.

Thanks!

Gabriel439 · April 13, 2023, 2:52am

One contributing factor to slow compile times that has come up repeatedly in the past is very large normal forms being serialized to the cache (e.g. ~/.cache/dhall), especially for projects that use dhall-kubernetes (since it has large normal forms, especially for certain imports). If that were the case then something like this would alleviate the problem:

github.com/dhall-lang/dhall-lang

RFC: Integrity check modes

opened 12:59AM - 09 Jun 21 UTC

Gabriella439

This is an idea inspired by https://github.com/dhall-lang/dhall-lang/issues/733 …and https://github.com/dhall-lang/dhall-haskell/issues/1890. I've mentioned this idea in passing in a couple of discussions and I'm doing a more detailed write-up here to try to make progress on this. The basic idea is to extend the integrity check syntax to provide a way for the user to optionally specify how much interpretation to apply to imported expressions before hashing and caching them. Given that the current interpretation phases before hashing (ignoring type-checking) are, in order: * Import resolution * β-normalization * α-normalization … then I propose the following extended syntax for integrity checks: ```dhall -- Perform no interpretation at all, **BUT** reject the expression if it has transitive imports that aren't -- protected by integrity checks -- -- This restriction ensures that the hash of the expression still unambiguously determines the result sha256:none:72a05… -- Resolve and type-check the expression, but don't β-normalize or α-normalize the expression sha256:resolve:72a05… -- Resolve, type-check, and β-normalize the expression, but don't α-normalize the expression sha256:beta:72a05… -- Resolve, type-check, and αβ-normalize the expression, the same as existing integrity checks: sha256:alpha:72a05… -- Initially a synonym for `sha256:alpha`, to capture user intent in case we add more phases (unlikely) sha256:full:72a05… -- Legacy integrity check, defined to be the same as `sha256:full:…` sha256:72a05… ``` I think each level of interpretation would be useful in its own right. For example: * `sha256:none` would (I believe) solve the performance issues described in https://github.com/dhall-lang/dhall-haskell/issues/1890 Specifically, if `dhall-kubernetes` and downstream projects used `sha256:none` everywhere then it would guarantee that large expressions (e.g. large types like `kubernetes.Resource`) would only be stored in the cache once and retrieved once. This would also make the Dhall cache much more like the `/nix/store` by permitting cache products to reference other cache products. * `sha256:resolve` would be useful if you wanted to preserve the original code as much as possible but you didn't want to have to protect all transitive imports with integrity checks * `sha256:beta` would be useful if you wanted to remove indirection but preserve variable names i.e. imports cached in this way won't have variables renamed to `_` when you fetch them from cache. I could see this integrity check mode being very useful for the Prelude, for example. * `sha256:alpha` would be the same as current integrity checks, useful if you want the hash to ignore all inessential differences i.e. if you want αβ-equivalent expressions to have the same hash Note that this would not break content-addressability of the cache: each `sha256` mode would still use the hash of the cache product as the lookup key. So, for example, if you saved the following expression to a file: ```dhall -- ./let.dhall let x = 1 in x ``` … and you protected it with an integrity check that specified no interpretation: ```dhall ./let.dhall sha256:none:33967e1bbb42a7058642c4a028fceeb6f295e92f97e48b35d18653fcd0ddfa26 ``` … then the first time the expression was resolved the interpreter would: * resolve `./let.dhall` * CBOR-encode the resolved expression as is (without any interpretation) * verify that the SHA256 hash of the CBOR-encoded expression matched the integrity check * store the CBOR-encoded expression underneath `${CACHE_DIRECTORY}/122033967e1bbb42a7058642c4a028fceeb6f295e92f97e48b35d18653fcd0ddfa26` … and subsequent resolutions of that same import would: * Fetch the cache product named `${CACHE_DIRECTORY}/122033967e1bbb42a7058642c4a028fceeb6f295e92f97e48b35d18653fcd0ddfa26` * Verify that the retrieved cache product matches the hash * Continue interpretation Moreover, this proposal would not require the creation of any new cache directories. Thanks to content-addressability the caches for each `sha256` mode could all coexist within the same cache directory and each mode could share cache products produced by other modes if they happened to coincide: To illustrate the latter point, suppose that you imported the following file: ```haskell -- example.dhall λ(_ : Type) → _ ``` … and you happened to specify no interpretation (even though the expression is already fully-interpreted): ```haskell ./example.dhall sha256:none:500579ecccc516ebba1c44a624aca77af683a718a5f7186510e1ad59f585180d ``` … then that would get cached under the `1220500579ecccc516ebba1c44a624aca77af683a718a5f7186510e1ad59f585180d` file within the cache directory. Then if you were to later change the file to: ```haskell -- ./example.dhall λ(a : Type) → a ``` … with a matching change to the integrity check: ```haskell ./example.dhall sha256:full:500579ecccc516ebba1c44a624aca77af683a718a5f7186510e1ad59f585180d ``` … then it would be a cache hit and the integrity check would accept the expression since it still matches the integrity check after the full interpretation.

… because it would allow opting out of storing normal forms so the cache products would be quicker to deserialize.

Does that sound similar to the issue that you’re having or do you suspect that it is something else?

Chris_Harris · April 13, 2023, 3:42am

Hello - thanks for the quick response - I’m building the dhall packages via nix (with buildDhallPackage, which I think already alpha-normalizes the source dhall expression before converting it to a binary: nixpkgs/build-dhall-package.nix at master · NixOS/nixpkgs · GitHub
so I’m not actually sure that this feature would make a difference.

Gabriel439 · April 15, 2023, 1:10am

Oh yeah, in that case that’s trickier to fix.

I still think the feature would make a difference because implementing that feature would also entail extending the cache to support expressions with unresolved import references (so long as they’re protected by an integrity check) (i.e. the sha256:none:… checks mentioned in that issue). Then you could extend the Nixpkgs support for Dhall to use those unresolved cache entries and sha256:none integrity checks to benefit from the performance improvement. However, all of this would be a lot of work which has not been standardized or accepted, so keep that in mind when setting expectations.

Chris_Harris · July 14, 2023, 7:20am

Hello - we’ve now reached the point at which our config fails to build (with an out-of-memory return code) on our nix build server, which has 64GB of memory. The parts of the config which seem to be causing trouble are not the raw data, but rather some of the functions which operate on the data (eg, to update a given element in a list). The config is still fine and works fast at evaluation-time.

I don’t mean to sound like I’m pressuring you to fix this situation, and I’d imagine there are no easy solutions, but we are trying to make a decision on what configuration system to use for the next version of the company’s product, and this problem makes it hard for me to push for Dhall. I find this unfortunate because Dhall fits our use case in many other ways and it’s a pleasant and well-thought-through language to use. So I’m not sure if there’s any useful outcome to me saying this, but I thought you might want to receive this feedback.