Now publishing pre-cached Prelude on IPFS

My CI for dhall-ruby now create a full cache directory of the standard Prelude with each build and publishes to IPFS.

You can browse this for example at https://gateway.pinata.cloud/ipfs/bafybeifugaluagsahc4xgipdrnn6cyq3pqd427guznemwjibzinb5tkvrq

This could be used to warm a cache with the entire Prelude so it never needs to be normalized locally, with the easiest way to do that manually being to download a tar using, for example, https://gateway.pinata.cloud/api/v0/get/bafybeifugaluagsahc4xgipdrnn6cyq3pqd427guznemwjibzinb5tkvrq and then untar the downloaded file and move the contents into your cache directory (if using the standard filesystem cache).

Another interesting use for this (especially if I start adding more libraries to my CI – suggestions welcome!) is that since IPFS is content-addressable, a given integrity-protected expression can be looked for in IPFS directly. For example using code like this:

def sha256toIPFS(sha256inHex)
  "/ipfs/b" + Base32.encode("\x01\x55\x12\x20" + [sha256inHex].pack("H*")).downcase.sub(/=*$/, '')
end

or this:

printf "/ipfs/b%s\n" $(echo 01551220$HEX_HASH | base16 -d | base32plain -l | sed -e's/=*$//')

The hex-encoded sha256 found in Dhall source code for an integrity-protected expression can be turned into a working IPFS path, which can be fetched even if you don’t have any local IPFS software like so:

https://gateway.pinata.cloud/ipfs/bafkreiabruu7amfulvscvotlxan7fqm2ppyyg2cgclhhulek7uqjs6b4ja

This could allow implementations to optionally look up in a sort of global “cache” on the fly for any integrity protected expression in lieu of resolving and normalizing it. So long as the implementation checks that the hash of the downloaded content matches the expected hash, this is just as safe (and often faster) as the normally used resolve-and-normalize mechanisms.

This works today for anything in the v9.0.0 Prelude.

2 Likes

This is really cool! Great work :slight_smile:

With a little work this could be made to work out-of-the-box for a vanilla Dhall interpreter using a virtual filesystem that essentially pretends to be ~/.cache/dhall (or any other directory if you set XDG_CACHE_HOME). Any time you read a file from the virtual file system it does an IPFS lookup and returns that and any time you write to it then it does ipfs add