Weird broken hashes for Prelude/JSON/Type

#1

I’m trying to freeze an import for Prelude/JSON/Type but I keep seeing strange behaviour:

$ dhall hash <<<https://prelude.dhall-lang.org/v16.0.0/JSON/Type
sha256:40edbc9371979426df63e064333b02689b969c4cfbbccfa481216d2d1a6e9759

$ dhall <<<'https://prelude.dhall-lang.org/v16.0.0/JSON/Type sha256:40edbc9371979426df63e064333b02689b969c4cfbbccfa481216d2d1a6e9759'
dhall: 
↳ https://prelude.dhall-lang.org/v16.0.0/JSON/Type sha256:40edbc9371979426df63e064333b02689b969c4cfbbccfa481216d2d1a6e9759

Error: Import integrity check failed

Expected hash:

↳ 40edbc9371979426df63e064333b02689b969c4cfbbccfa481216d2d1a6e9759

Actual hash:

↳ c8675d83d1accca06409a48308b85fafa40a07a7c024422bdc6d3c2396484622


1│ https://prelude.dhall-lang.org/v16.0.0/JSON/Type sha256:40edbc9371979426df63e064333b02689b969c4cfbbccfa481216d2d1a6e9759

(input):1:1

This is consistent: when I check the hash with dhall hash (or when I try to freeze an import with dhall freeze) I get the 40edb... hash; but if I try to use an import with that hash protecting it, I get the import failure.

Does anyone know where to begin debugging this? :confused: Why does dhall hash think the import has one hash, but dhall itself think this hash is incorrect?

#2

Strange!

$ echo 'https://prelude.dhall-lang.org/v16.0.0/JSON/Type sha256:40edbc9371979426df63e064333b02689b969c4cfbbccfa481216d2d1a6e9759' | dhall

works fine for me.

Which version of dhall are you using?

Does anything change if you use the --no-cache option?

#3

I’m running 1.31.1. If I use --no-cache it works.

Maybe dhall-golang has put an incorrect entry in the cache (until recently, it didn’t alpha-normalize things correctly before caching them). Something to investigate on my end.

#4

After further debugging, it seems that:

  • bugs in older & development versions of dhall-golang put bad cache entries in my cache
  • dhall-haskell 1.31.1 barfed on them; the presence of these bad cache entries prevented dhall-haskell from trying to fetch the upstream resource

Obviously dhall-golang was wrong to put the bad cache entries in place. But I think dhall-haskell is also wrong to blow up on a bad cache entry, when it can always go to the upstream and pull down a fresh copy.

I’m hoping that the test I added in https://github.com/dhall-lang/dhall-lang/pull/983 fixed this problem for dhall-haskell 1.32.0. I can’t see anything to that effect in the release notes, though, and I can’t upgrade via nix right now to try it myself because of the current hydra problems.

#5

Thanks for the explanation @philandstuff. Looks like dhall-haskell needs a fix then.

To be clear: dhall should have downloaded the file, and overwritten the incorrect content for sha256:40edbc9371979426df63e064333b02689b969c4cfbbccfa481216d2d1a6e9759 in the cache, right?

#6

I’m surprised if dhall 1.32.0 isn’t fixed, because the test in the linked PR should have caught this bug. Is dhall-haskell running the test-import tests correctly?

EDIT: also, yes, I think you are correct, I think dhall should have ignored the bad cache file and then overwritten with good content from the download.

#7

I’m not sure.

≻ cabal test dhall:tasty --test-options="-p IgnorePoisoned" --test-show-details=streaming -w ghc-8.6.5 -O0
Build profile: -w ghc-8.6.5 -O0
In order, the following will be built (use -v for more details):
 - dhall-1.32.0 (test:tasty) (first run)
Preprocessing test suite 'tasty' for dhall-1.32.0..
Building test suite 'tasty' for dhall-1.32.0..
Running 1 test suites...
Test suite tasty: RUNNING...
Dhall Tests
  import tests
    discover
      ./dhall-lang/tests/import/success/unit/IgnorePoisonedCacheA.dhall: OK (0.03s)

All 1 tests passed (0.03s)
Test suite tasty: PASS

But I don’t see any updates corresponding to the new instructions in https://github.com/dhall-lang/dhall-lang/pull/983.

v1.32.0 still barfs on incorrect cache content in any case.

#8

The instructions haven’t changed (other than to clarify the test cache is read-only), but maybe dhall-haskell was incorrect before.

You can test this behaviour manually by changing directory to dhall-lang/tests/import/success/unit/ and running:

XDG_CACHE_HOME=../../cache/ dhall < IgnorePoisonedCache

Setting XDG_CACHE_HOME causes dhall to use the test cache with the poisoned entry.

If you are running the tests/import tests correctly, you should be using the tests/import/cache/dhall cache, as documented in tests/README.md.

#9

I’ve filed this in https://github.com/dhall-lang/dhall-haskell/issues/1790 and https://github.com/dhall-lang/dhall-haskell/issues/1791 for now.

1 Like