Best practises around `dhall freeze`

#1

First of all thanks to all contributors for creating such an inspiring project as dhall!

I got a couple of questions about dhall freeze, I will enumerate all of them here as opposed to opening a few questions. I hope that’s fine.

  1. a) Is there any way to dhall freeze in place, i.e. that the input and oputput file is the same? I tried out dhall freeze --ascii < a.dhall > a.dhall but it results in Error: Invalid input and a.dhall being stripped to 0 bytes. I ended up creating some temporary files but it’s a bit involved.

  2. b) I noticed --inplace mentioned somewhere but I think it was only a suggestions that never materialized - I tried --inplace for both format and freeze and it didn’t work. Do you think that would be a useful addition? I think code formatters usually offers such funcionality as this is what you usually want to do on CI or as part of git hook.

  3. I noticed that dhall freeze also formats the code. I know it may feel natural but I was not expecting that - I wanted to only freeze imports without formatting. I am curious if that’s intentional and if that should be mentioned somehow in e.g. dhall --help message?

  4. Is there some specification on how dhall freeze is supposed to work? I have “frozen” all my imports and run dhall freeze once more, this time without internet connection, and it was still trying to resolve “frozen” imports. I am curious whether it means that even “frozen” (i.e. ones with hash) still can be resolved once more?

  5. Are there any best practises around how one should use dhall freeze in the codebase? Example question - I just created a new dhall file in my project and it imports Prelude. Should I copy and paste hash from previous imports in that project in order to be consistent
    across files within one project?

#2

On (1) and (2): dhall freeze --inplace FILE should definitely work. If it doesn’t, please make a bug report at https://github.com/dhall-lang/dhall-haskell/issues. Same for dhall format.

Regarding (3): Modifying code while preserving the formatting is actually pretty tricky and (in the current implementation) more work than using the existing pretty-printer. We have an open feature request though: https://github.com/dhall-lang/dhall-haskell/issues/1356 Feel free to chime in there!

(4) dhall freeze ignores any existing hashes, and resolves imports afresh. It’s not transitive in the sense that imports in imports are resolved again. There some discussion regarding this in https://github.com/dhall-lang/dhall-haskell/issues/1551.

(5) All imports of the same “thing” should of course have the same hash. But freeze should take care of that on its own. Copying by hand shouldn’t be necessary. Regarding ergonomics, especially freezing an entire project in one sweep, see https://github.com/dhall-lang/dhall-haskell/issues/1551 again.

2 Likes
#3

@note: I suspect the problem you ran into for dhall-freeze might be a bug with optparse-applicative because it appears to not work if you specify the --ascii flag first:

$ dhall freeze --ascii --inplace ./example.dhall 
Invalid option `--inplace'

Usage: dhall ([version] | [resolve] | [type] | [normalize] | [repl] | [diff] |
             [hash] | [lint] | [tags] | [format] | [freeze] | [encode] |
             [decode] | [text] | [to-directory-tree] | [--file FILE]
             [--output FILE] [--annotate] [--alpha] [--no-cache] [--version])
             [--explain] [--plain] [--ascii] [--censor]
  Interpreter for the Dhall language

… but if you specify the --ascii flag afterward then it should work:

$ dhall freeze --inplace ./example.dhall --ascii
#4

Thank you!

It took me a while to come up with correct order but the following worked:

dhall --ascii freeze --inplace functions.dhall

I was trying out dhall freeze --ascii --inplace functions.dhall but then I was getting invalid option error. No idea if that can be regarded a bug or is it rather incorrect usage.

There’s a problem with discoverability of CLI in general. I am thinking about writing a list of most commonly used commands and documenting parameters more thoroughly than --help does. Do you think approaching it here https://github.com/dhall-lang/dhall-haskell/issues/537 would be a good idea?

I opened an issue for lack of --inplace in output of --help.

As I was composing that comment @Gabriel439 mentioned alternative command syntax - it works too!

@sjakobi regarding (3), (4), (5) - thanks for the links, they cleared that out for me

#5

I’d definitely use man pages if they were available.

#6

@note @sjakobi @Gabriel439 what does --ascii means in the context of freezing ?

I have just recently realized that freezing makes a big difference regarding performance for the dhall package clients. On the other hand, it is a pain while developing the dhall package. Is there a way to unfrozen a file with the CLI ?

Actually it looks like “freezing” is a huge boost on the client side - the code that import the package.dhall file - but it does not seem to make much difference at all (in term of performance) to freeze every imports within the project. Is that correct ?

#7

@PierreR: There shouldn’t be a need to freeze local files, if that’s what you are suggesting. The Haskell implementation already implicitly caches local imports (even those without an integrity check).

#8

@Gabriel439 Not sure how I can comprehend such figure then.

Given this file:

let oc =
      ../package.dhall sha256:6ef4eacc29986073de97ea4c9aee36240690b1d3684eb59bf112d204ddda682b

let project =
      oc.Project::{
      , name = "cicd-doc-dev"
      , displayName = "cicd doc for docs.cicd.cirb.lan"
      , requester = "cicd"
      }

in  project


→ time dhall type --quiet <<< ./openshift/examples/project1.dhall
37.31s user 1.98s system 99% cpu 39.495 total

After removing the hash:

→ time dhall type --quiet <<< ./openshift/examples/project1.dhall
104.18s user 2.60s system 99% cpu 1:47.31 total

Such figures are obtained for similar files in a regular/reproducible manner. I have checked and all the remote transitive imports used are frozen.

#9

@PierreR: I can’t say for sure unless I can reproduce locally

#10

@Gabriel439 I have exposed the repository here:

In ‘openshift/examples’ (which are examples of client codes), removing the sha makes a great difference performance wise.

Without freezing the examples:

→ time make examples                  
dhall-to-yaml --file openshift/examples/project1.dhall > openshift/examples/project1.yaml
dhall-to-yaml --file openshift/examples/application1.dhall > openshift/examples/application1.yaml
dhall-to-yaml --file openshift/examples/cron1.dhall > openshift/examples/cron1.yaml
dhall-to-yaml --file openshift/examples/secret1.dhall > openshift/examples/secret1.yaml
make examples  358.39s user 5.80s system 99% cpu 6:04.96 total

After adding the sha in each of the 4 examples:

→ time make examples               
dhall-to-yaml --file openshift/examples/cron1.dhall > openshift/examples/cron1.yaml
dhall-to-yaml --file openshift/examples/secret1.dhall > openshift/examples/secret1.yaml
dhall-to-yaml --file openshift/examples/project1.dhall > openshift/examples/project1.yaml
dhall-to-yaml --file openshift/examples/application1.dhall > openshift/examples/application1.yaml
make examples  81.43s user 2.36s system 99% cpu 1:23.90 total
#11

@PierreR: Never mind. What I said was wrong: the implicit cache of the Haskell implementation still has to at least resolve transitive imports, so it’s not necessarily going to be as fast as a frozen import.

1 Like