Versioning Dhall types

(Philip Potter) #1

This is maybe a slightly vague question because I haven’t yet worked out all my thoughts, but:

I’m thinking about how I might configure an app which consumes Dhall natively. I have my Dhall file, and everything works on app 1.2, say.

Now let’s say app 1.3 is released. There’s a new feature which requires a new field in the config type. This means that 1.2 and 1.3 have different config types.

So if I have fully normalised my Dhall, my 1.2 Dhall file won’t typecheck correctly as a 1.3 file. I don’t want to use the new feature, I just want to upgrade.

How do I use my old config file with the new app version?

There may be ways to solve this, eg by passing the type and defaults into the config file. But who passes them in?

This compares unfavourably with yaml configuration where upgrades that bring new features just work with old config files.

What are the best practices around versioning Dhall config types? In particular, how do I allow old config files to work with new app versions with as little work as possible?

(Ari Becker) #2

Currently, I think best practice here is to host the types in a network-accessible location, and use that when trying to import a configuration which matches that type. Think of something like the following pseudocode:

imported_version = null
config = null
for version in 1.3, 1.2:
    config = import "./config.dhall : (${version}/types.dhall).Config"
    imported_version = version
  printerr "error: config import failed, trying version type ${version}"
if config == null || imported_version == null : raise Exception "couldn't import config"

And then use the imported_version to, for example, tell the program whether to enable or disable the new feature.

The issue with doing this internally to Dhall is if the old config is missing a field that the new type requires, clearly such an import should completely fail. The question is what should happen if the additional field is optional, which is something that, to my understanding, is under active discussion.

(Gabriel Gonzalez) #3

We have a similar situation at work where we distribute some data via Dhall configuration files to machines running different releases of our product. Each release of our product can potentially expect a different schema for the Dhall configuration file that they receive.

The way we do this is that each appliance checks for a Dhall configuration file named after its own product version (i.e. ./2.0/Config.dhall, for example) and a default fallback configuration file that it checks if there is no version-specific configuration (i.e. ./default/Config.dhall). In our case the fallback configuration file is the one for the latest version of the product (i.e. ./latest/Config.dhall)

Additionally, you can define configuration files for older versions in terms of newer versions so that you retain a single source of truth. For example, suppose that our schema changed over time like this:

-- ./1.0/Type.dhall

{ foo : Natural }
-- ./2.0/Type.dhall

{ foo : Natural, bar : Bool }
-- ./3.0/Type.dhall

{ foo : Optional Natural, bar : Bool }

We can define a configuration file and schema for our latest release (3.0):

-- ./latest/Config.dhall

{ foo = Some 1, bar = True } : ./Type.dhall
-- ./latest/Type.dhall

{ foo : Optional Natural, bar : Bool }

Then we can specify that version 3.0 of our configuration is identical to the latest one if we want to be explicit and not rely on the fallback

-- ./3.0/Config.dhall

../latest/Config.dhall : ./Type.dhall
-- ./3.0/Type.dhall


In version 2.0, foo was a required field, so we can define version 2.0 in terms of version 3.0 if we define a suitable fallback value for the foo field:

-- ./2.0/Config.dhall

let newer = ../3.0/Config.dhall

in      newer.{ bar }
      ⫽ { foo = Optional/fold Natural Natural (λ(n : Natural) → n) 0 }
    : ./Type.dhall

Ideally, we’d also be able to define ./2.0/Type.dhall in terms of ./3.0/Type.dhall, but Dhall currently does not support projecting out fields from a record type like it does for record values. Maybe it should?

-- ./2.0/Type.dhall

{ foo : Natural, bar : Bool } 

In version 1.0, there was no bar field, so we can trivially project out just the foo field from version 2.0:

-- ./1.0/Config.dhall

let newer = ../2.0/Config.dhall in newer.{ foo } : ./Type.dhall
-- ./1.0/Type.dhall

{ foo : Natural }

And now, any time we make a change to ./latest/Config.dhall, it will get correctly “backported” to older support configurations:

$ dhall <<< './3.0/Config.dhall' 
{ bar = True, foo = Some 1 }
$ dhall <<< './2.0/Config.dhall' 
{ bar = True, foo = 1 }
$ dhall <<< './1.0/Config.dhall' 
{ foo = 1 }

While it is more verbose than YAML’s support for ignoring absent fields it can handle some things that YAML can not such as defaulting values that transition from optional to required or more drastic schema changes like renaming fields or refactoring record hierarchies. You have much greater freedom to change your schema when you can express data migrations within the language.

(Philip Potter) #4

Thanks, that’s really interesting! I hadn’t seen that pattern before.

Unfortunately I think it’s somewhat orthogonal to my use case. I’m thinking from the point of view of third-party software (in a GitHub issue I mentioned Alertmananger, but it could be anything similar).

Here’s my concern: I want to be able to pull in point-release and minor-release upgrades as easily as possible. Assuming a semver model, point releases can’t change config types, but minor releases can.

With traditional config file formats, I can just run apt-get upgrade (or nix-env -u or whatever else) and every piece of third party software I’m running gets upgraded, keeping the same configuration as before.

Any solution which requires me to edit configuration files to do a minor release upgrade feels like unnecessary toil. With a sysadmin hat on, I operate a lot of software and I expect it to endeavour to make upgrades as frictionless as possible.

I may have misunderstood but I still don’t see how I get from Gabriel’s solution to this frictionless upgrade outcome. It seems that if I want an old config file to work with a new version, I have to edit the config file.

I think Ari’s idea might work for me though - allowing (for example) alertmanager 1.3 to accept config that is typed as 1.3, 1.2, 1.1 or 1.0 makes a lot of sense and achieves what I want, possibly at the expense of added internal complexity for the alertmanager server itself. Then I can use the same config file, without edits, before and after an upgrade.

(Gabriel Gonzalez) #5

@philandstuff: If you control the app being upgraded, you can have the application supply any defaults that the user does not specify by doing:

appDefaults // userConfig

… in other words, the application bakes in default values for all fields and the user only specifies how their configuration differ from the default. If you add new fields (which the user doesn’t use) then this requires no change to the user’s configuration. Each version of your application supplies appDefaults appropriate to its own version when interpreting the above expression.