Modelling AWS Cloudformation `Ref` function

#1

AWS Cloudformation is a configuration language which allows users to specify and deploy resources on AWS.

The number of available resources and the number of configuration options for each is staggering.

Since the configuration is in JSON or YAML it suffers from the usual issues around discoverability and type safety (there are lots of options which require specific strings and lots of situations where attributes should be provided together i.e. we can use types to make configuration errors impossible).

Amazon themselves have recognized that simply getting started with Cloudformation is an intimidating prospect and have created SAM which includes a macro to simplify the specification of resources and their implicit dependencies. Macros too are specified in JSON.

In short, Cloudformation seems like a clear use case for Dhall!

Since I’m new to Dhall I first wanted to ask: has anyone worked on this already? Googling brought up this blog but the author noted some problems they ran into and doesn’t appear to have moved forward with Dhall.

If there is not an existing implementation, I would also like some guidance on how to model the aspects of the language, particularly those which represent implicit computations / sequencing.

Declared resources may depend on the identity of other resources using the Ref function. This allows the Cloudformation execution service to determine the dependency graph of resources, sequence the creation of resources and substitute the returned values into subsequent resource definitions.

How would I go about representing this in Dhall? Specifically, is there a way to do this which has a degree of type safety?

Thanks,

Michael

2 Likes
#2

@enetsee: You will probably want to do something similar to what dhall-kubernetes does and auto-generate Dhall types from a CloudFormation Resources specification. I’m not aware of any project that has done this, yet.

One of the issues mentioned in that blog can be addressed by the fact that dhall-to-{yaml,json} support the JSON type from the Prelude, meaning that you can stick the JSON type anywhere in your configuration where you don’t know what the schema for the JSON will be (i.e. if you want to be able to store arbitrary JSON). This allows you to have a “gradually typed” schema where you stick JSON anywhere that you don’t have a more precise type.

One tricky bit here is support CloudFormation functions. I’m not an expert on CloudFormation, but my understanding is that anywhere CloudFormation expects a string it could be an ordinary JSON string or it could be a CloudFormation “expression” that returns a string, such as:

{ "Fn::Base64" : "foo" }

… and I believe you can chain those functions, too:

{ "Fn::Base64" : { "Fn::Base64" : "foo" } }

However, this is possible to model in Dhall, especially once we support forward declarations. You can either wait for that feature to be standardized or you can get started with the equivalent feature using lambdas, which I will outline briefly here.

The idea is that each CloudFormation template would begin by “declaring” a dependence on outside types and builtins, like this:

    \(CloudFormationText : Type)  -- `Text` that could be produced by a CloudFormation function
->  \(PlainText : Text -> CloudFormationText)  -- Plain `Text`
->  \(Base64 : CloudFormationText -> CloudFormationText)  -- Fn::Base64
->  \(Join : CloudFormationText → List CloudFormationText → CloudFormationText)  -- Fn::Join
->  …

… and you can add additional CloudFormation “types” and “functions” in this way. Using the forward declarations feature I linked to above, that would become:

let CloudFormationText : Type

let PlainText : Text -> CloudFormationText

let Base64 : CloudFormationText -> CloudFormationText

let Join : CloudFormationText → List CloudFormationText → CloudFormationText

…

Then anywhere in the CloudFormation schema where it expects a string that could be generated by functions, you can have it expect a CloudFormationText instead. Then you could write:

{ foo = Base64 (PlainText "foo")
}

Then I believe you could convert such a strongly-typed schema that the user authors to the JSON type. In other words, there could be a render : CloudFormationSchema → JSON function implemented in Dhall that performs this conversion. Then it would take care of supplying the implementations of the CloudFormationText/PlainText/Base64/Join/… builtins that the CloudFormationSchema expects. In other words, the CloudFormationSchema type would be something like:

    forall (CloudFormationText : Type)
->  forall (PlainText : Text -> CloudFormationText)
->  forall (Base64 : CloudFormationText -> CloudFormationText)
->  forall (Join : CloudFormationText -> List CloudFormationText -> CloudFormationText)
->  ...  -- Then the strongly-typed schema of the CloudFormation resource type

… and the render function would look something like this:

let CloudFormationSchema = ./CloudFormationSchema.dhall

let JSON = https://prelude.dhall-lang.org/JSON/package.dhall

let CloudFormationText = JSON.Type

let PlainText = JSON.string

let Base64 =
        \(input : CloudFormationText)
      -> JSON.object (toMap { `Fn::Base64` = input })

let Join =
        \(delimiter : CloudFormationText)
      -> \(values : List CloudFormationText)
      -> JSON.object (toMap { `Fn::Join` = JSON.array [ JSON.string "delimiter", JSON.array values ] })

let render : CloudFormationSchema -> JSON.Type =
          \(config : CloudFormationSchema)
      ->  let applied = config CloudFormationText PlainText Base64 Join ...

          in  ... -- Further conversion logic from the strongly typed schema to the weakly typed schema

Note that the trick I’ve described is essentially the same principle as the trick illustrated in How to translate recursive code to Dhall

2 Likes