First steps failed when trying dhall for gitlab-ci

dhamidi · July 15, 2022, 12:04pm

Hey everyone,

I’m new here, working together with @Lee_Hambley. We both really like dhall and specifically want to use it to generate our Gitlab CI YAML files, because they have reached a level of complexity that is hard to deal with in plain YAML.

Unfortunately, my first experience with dhall is frustrating. Here is the series of steps I’ve been through:

Learn about dhall and get excited
Install dhall
Search for a dhall-based solution to my problem and find it https://github.com/bgamari/dhall-gitlab-ci
Copy-paste the example from the README and hit a wall

why does the import fallback in the Prelude.dhall not work?

Context

the import integrity check fails when importing the Gitlab module, but I don’t see the reason why:

:; export DHALL_PRELUDE='https://prelude.dhall-lang.org'
:; dhall-to-yaml-ng --file .gitlab-ci.dhall


↳ https://raw.githubusercontent.com/bgamari/dhall-gitlab-ci/master/package.dhall
  ↳ https://raw.githubusercontent.com/bgamari/dhall-gitlab-ci/master/GitLab/package.dhall
    ↳ https://raw.githubusercontent.com/bgamari/dhall-gitlab-ci/master/GitLab/Job/package.dhall
      ↳ https://raw.githubusercontent.com/bgamari/dhall-gitlab-ci/master/GitLab/Job/Type.dhall
        ↳ https://raw.githubusercontent.com/bgamari/dhall-gitlab-ci/master/GitLab/Prelude.dhall
          ↳ https://raw.githubusercontent.com/bgamari/dhall-gitlab-ci/master/Prelude.dhall
            ↳ https://raw.githubusercontent.com/dhall-lang/dhall-lang/v17.0.0/Prelude/package.dhall
  sha256:0c04cbe34f1f2d408e8c8b8cb0aa3ff4d5656336910f7e86190a6d14326f966d

Error: Import integrity check failed

Expected hash:

↳ 0c04cbe34f1f2d408e8c8b8cb0aa3ff4d5656336910f7e86190a6d14326f966d

Actual hash:

↳ 10db3c919c25e9046833df897a8ffe2701dc390fa0893d958c3430524be5a43e


24│   https://raw.githubusercontent.com/dhall-lang/dhall-lang/v17.0.0/Prelude/package.dhall sha256:0c04cbe34f1f2d408e8c8b8cb0aa3ff4d5656336910f7e86190a6d14326f966d

This is also detailed in this issue on Github: https://github.com/bgamari/dhall-gitlab-ci/issues/13

But the Prelude.dhall file explicitly has a fallback without the integrity check:

  env:DHALL_PRELUDE
? https://raw.githubusercontent.com/dhall-lang/dhall-lang/v17.0.0/Prelude/package.dhall sha256:0c04cbe34f1f2d408e8c8b8cb0aa3ff4d5656336910f7e86190a6d14326f966d
? https://raw.githubusercontent.com/dhall-lang/dhall-lang/v17.0.0/Prelude/package.dhall

What happened

Setting DHALL_PRELUDE apparently had no effect, so it fell back to the first import, which fails due to the failed integrity check and then execution stops.

What I expected to happen

The final import without the integrity check is executed and works, since there is no integrity check.

Gabriel439 · July 15, 2022, 3:06pm

This sounds pretty similar to:

github.com/dhall-lang/dhall-haskell

Output of "dhall freeze --cache" no longer works with Dhall v21.0.0

opened 04:28PM - 08 Dec 21 UTC

closed 04:38PM - 15 Dec 21 UTC

joell

The `dhall freeze` command has a `--cache` flag thats described purpose is to "A…dd fallback unprotected imports when using integrity checks purely for caching purposes". However, in dhall-haskell 1.40.1 the code generated from using this feature no longer works if the fetched URL content does not match the hash. Example: ```sh # observe how dhall freeze creates a cache-only fallback import for List/map $ dhall freeze --cache <<< 'https://prelude.dhall-lang.org/List/map' https://prelude.dhall-lang.org/List/map sha256:dd845ffb4568d40327f2a817eb42d1c6138b929ca758d50bc33112ef3c885680 ? https://prelude.dhall-lang.org/List/map # submit the same code to Dhall but with a hash that does not match the content $ echo ' https://prelude.dhall-lang.org/List/map sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff ? https://prelude.dhall-lang.org/List/map' | dhall dhall: ↳ https://prelude.dhall-lang.org/List/map sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff Error: Import integrity check failed Expected hash: ↳ ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff Actual hash: ↳ dd845ffb4568d40327f2a817eb42d1c6138b929ca758d50bc33112ef3c885680 1│ https://prelude.dhall-lang.org/List/map sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff (input):1:1 ``` ~~The expectation I had from the description of `dhall freeze --check` was that the `?` operator should fallback from the integrity-check-failing import to the unprotected URL import and the expression should have succeeded (as long as https://prelude.dhall-lang.org/List/map could be loaded). Put simply, it seems this should work without emitting an error message.~~ ~~Is `dhall freeze --check`'s description out of date, dhall-haskell's implementation buggy, or my interpretation of all this faulty?~~ (**Update**: Given the breaking change to `?` in Dhall v21.0.0, it seems the issue here is in the output of `dhall freeze --cache` not being revised to adapt to the breaking change.)

… which was fixed in version 1.41.0 or later of the Haskell implementation of Dhall. Double-check that you’re using that version or newer.

The environment variable import only works for files that were imported locally (i.e. not GitHub repositories), which is why that workaround did not work. Something like env:DHALL_PRELUDE ? … is only useful for local development.

Gabriel439 · July 15, 2022, 3:11pm

Oh wait, sorry. I was mistaken. That bug and resolution fixed the output of dhall freeze --cache, but didn’t change how imports were actually resolved.

The actual issue here is that in version 21.0.0 of the language standard the ? operator no longer falls back in the event of a hash mismatch so the idiom of https://foo sha256:… ? https://foo is no longer correct. While the output of dhall freeze --cache was changed to adjust, existing repositories in the wild (such as dhall-gitlab-ci) that used the old idiom are now broken in the way you just described.

The fix here is that you would need to submit a patch to amend the Prelude.dhall file to:

  env:DHALL_PRELUDE
? missing sha256:0c04cbe34f1f2d408e8c8b8cb0aa3ff4d5656336910f7e86190a6d14326f966d
? https://raw.githubusercontent.com/dhall-lang/dhall-lang/v17.0.0/Prelude/package.dhall

… and probably also fix the hash along the way

dhamidi · July 15, 2022, 3:51pm

First of all, thank you for the prompt reply!

A change of the behavior of ? explains the problem, thank you!

I’m on a very recent version of dhall:

:; dhall --version
1.41.1

The gitlab-ci repository looks pretty inactive. We’ll see how far we take this at work and might publish a fork with all of the fixes applied if we end up using it enough.

I’ve linked your answer to the issue on Github

bsaul · July 18, 2022, 3:42pm

Gabriel439:

The fix here is that you would need to submit a patch to amend the Prelude.dhall file to:
  env:DHALL_PRELUDE
? missing sha256:0c04cbe34f1f2d408e8c8b8cb0aa3ff4d5656336910f7e86190a6d14326f966d
? https://raw.githubusercontent.com/dhall-lang/dhall-lang/v17.0.0/Prelude/package.dhall
… and probably also fix the hash along the way

Indeed. That’s we had to do.

Here’s a Prelude.dhall where that’s fixed.