Deep imports and environment variables

#1

I’m running into something a bit surprising

I have a setup for creating Docker Compose files (YAML) using Dhall and I’ve put the various bits of configuration into each service’s repo on GitHub. The repos are private so I have a token and I import using

let svcA = https://raw.githubusercontent.com/TheOrg/svcARepo/master/service.dhall
          using [ { header = "Authorization" , value = "token ${env:GITHUB_TOKEN as Text}" } ]

the issue I run into is that if the expression for svcA itself tries to import another service, in the same manner, then I seem to get the following error

↳ ./service.dhall

  ↳ https://raw.githubusercontent.com/TheOrg/svcB/master/service.dhall using [ { header = "Authorization", value = "token [secure]" } ]

Referentially opaque import: env:GITHUB_TOKEN as Text

2│                                                                 env:GITHUB_TOKEN as Text

Is it not possible to import using headers containing envvars in several levels like this?

Supporting transitive imports from private repositories?
#2

It seems like excepted according to: https://docs.dhall-lang.org/discussions/Safety-guarantees.html#cross-site-scripting-xss

1 Like
#3

Ah, too bad, because this is the central piece of my use case – keeping expressions relating to a service in its own repo – so I really need to do deep imports while keeping the repositories private.

Is there a way to turn this off?

#4

There are really good security and sanity reasons for why a remote import cannot access environment variables. I can suggest the following alternatives:

  1. if there’s nothing sensitive in the dhall files, pull them out into separate, public, repos, so that you don’t need the github token
  2. require the user to clone the repos locally, and have the service.dhall files import each other locally rather than off of raw.githubusercontent.com
  3. use relative imports (so, from within svcA’s service.dhall file, rather than importing https://raw.githubusercontent.com/TheOrg/svcB/master/service.dhall, import ../../../TheOrg/svcB/master/service.dhall). This works because it is on the same domain and so will use the same headers. It’s a bit of a hack though, and it won’t work the same way locally because the path structure isn’t quite the same as local filesystem would be (it has an extra level for the branch name).
  4. I don’t know if dhall supports a .netrc file, but if it does, this might be a way to specify your authentication without the remote import being able to see your github token?

I don’t love any of these options but I think at least one of them should be able to get you going again?

#5

Yes, I thought of the relative import as well, so now I have three-level imports everywhere

  1. relative path that works on GitHub (I get failures if this isn’t the first one, which is a bit strange, I think)
  2. relative path that works locally (for local development and speed)
  3. GitHub URL (for use locally to not require checking out the repo)

It does seem to work.

#6

@magthe: There’s another option, which is to make the code a function of what they need instead of depending on an environment variable. In other words, instead of this:

1 + env:EXAMPLE

… you do this:

λ(EXAMPLE : Natural) → 1 + EXAMPLE

Then you only need to import once at the top-level of your program and thread the imported value through function arguments to where it is needed

#7

I was considering that, however, since I really want each repo to only care about its own immediate dependencies it won’t be acceptable in this particular case.

I did try to pass in the token:

\ (token : Text) ->
  let imp = https://<github URL>
            using [ { header = "Authorization" , value = "token ${token}" } ]
...

but that didn’t work. I do get the reasons for why it failed, though in this particular case it failed with a token is undefined, which I would argue isn’t the best of error messages.

#8

We achieve a similar usecase in Concourse. Instead of referring to the http://raw.githubusercontent.com/owner/private-repo/ref/config.dhall directly, we use Concourse to retrieve the repository (repositories) into a container, and then refer to them with normal local-directory access patterns (./private-repo/config.dhall). As a bonus, Concourse manages the ref for us.