Records with truly optional fields

#1

My application uses the following Dhall type:

{ siteTitle :
    Text
, author :
    Optional Text
, siteBaseUrl :
    Optional Text
, editUrl :
    Optional Text
}

The users of my application would use the following as their configuration file (note that no type is specified for this value):

{ siteTitle =
    "Srid's Zettelkasten"
, author =
    Some "Srid"
, siteBaseUrl =
    None Text
, editUrl =
    None Text
}

Is there a way to have the users of the application ignore those None fields? So that they can instead specify:

{ siteTitle =
    "Srid's Zettelkasten"
, author =
    Some "Srid"
}

I would like this to be possible so that a future version of the application, which may add new fields (optional) to the configuration, would not break when reading existing user configuration (which would not have that field).

If this cannot be done Dhall, I’ll unfortunately have to switch to something else.

#2

You can use record completion to make this nice for end users, e.g.

let SiteInfo =
      { Type =
          { siteTitle : Text
          , author : Optional Text
          , siteBaseUrl : Optional Text
          , editUrl : Optional Text
          }
      , default =
          { author = None Text
          , siteBaseUrl = None Text
          , editUrl = None Text }
      }

in  SiteInfo::{ siteTitle = "My Site" }

Record completion lets you specify default values for fields, so fields you are okay with being None by default you don’t have to specify. (And this lets you add new fields in the future without breaking extant configurations, as long as they have defaults.)

4 Likes
#3

See also:

2 Likes
#4

Interesting. But the problem is that users won’t have access to the SiteInfo type. Their config file will contain only SiteInfo::{ siteTitle = "My Site" }. So how do we bring SiteInfo into scope? Can this be done with input auto ... of dhall-haskell?

I could hack things by doing this:

      config <- readFile' $ toFilePath configPath
      let configFull = "let Neuron = ./src-dhall/Neuron.dhall in " <> config
      liftIO $ input auto (toText configFull)

Proof of concept

But this will work only on the development machine. Users won’t have access to src-dhall/....

#5

I might not be understanding exactly what your goal is here, but can’t you put your types on github or whatever? So the user can just do like

let SiteConfig = https://raw.githubusercontent.com/srid/dhall-site-config/master/SiteConfig {- Or whatever it is. -}

in SiteConfig::{ siteTitle = "My Site" }
#6

This forces the users to load an external file when the file in question could be available locally. Local import might also be difficult because you might not know the location easily.

For my own project I was thinking of making the user configuration be a function, and then injecting the dependencies to it using the host language.

\(SiteConfig : SiteConfig) -> SiteConfig::{ siteTitle = "My site" }

This is what nix does in many places.

I did not implement this yet myself. I don’t know if this is easy to do or not. I’d be curious to see how this could be done.

#7

For what it’s worth, one way of working around this is to use json-to-dhall.

example.json:

{
  "siteTitle" : "Srid's Zettelkasten",
  "author" : "Srid"
}

Config.dhall:

{ siteTitle : Text
, author : Optional Text
, siteBaseUrl : Optional Text
, editUrl : Optional Text
}

in a terminal:

$ json-to-dhall ./Config.dhall --file ./example.json
{ author = Some "Srid"
, editUrl = None Text
, siteBaseUrl = None Text
, siteTitle = "Srid's Zettelkasten"
}

This workaround only works as long as you don’t keep functions in your configuration type, i.e.

{ siteTitle : Text
, author : Optional Text
, siteBaseUrl : Optional Text
, editUrl : Optional Text
, webhookTemplate : Text -> Text
}

For support without a workaround, subscribe to the issue for row polymorphism: https://github.com/dhall-lang/dhall-lang/issues/828

#8

I agree, this seems like an ideal solution. As @ari-becker pointed out however, this is currently impossible in dhall, since there is no type you could give to SiteConfig here that would work.
We would need row polymorphism to make that work. Then users would write:

\(SiteConfig : { Type : Type, default : Row Type }) -> SiteConfig::{ siteTitle = "My site" }
#9

Actually, since you appear to be using the Haskell lib directly, a simple solution would be to define SiteConfig as a custom builtin value. Users would only have to write

SiteConfig::{ siteTitle = "My site" }

See here for how to do that.

1 Like
#10

If the type is

let SiteInfo =
      { Type =
          { siteTitle : Text
          , author : Optional Text
          , siteBaseUrl : Optional Text
          , editUrl : Optional Text
          }
      , default =
          { author = None Text
          , siteBaseUrl = None Text
          , editUrl = None Text }
      }

then it would work, right?

Edit: What I mean to say is that in this case we are dealing with a concrete type and there should be no need to be polymorphic.

#11

Oh yeah I mixed things up. I meant:

\(DefaultType : Row Type) -> \(SiteConfig : { Type : Type, default : DefaultType }) -> SiteConfig::{ siteTitle = "My site" }

Because what we want is to not have to mention the actual type in the user’s code, so that the user code can stay the same when the library adds more fields.

#12

Why would you leave the user with no clue about the type? I mean, when I configure an application I always look for linters and such, and having the type is the way the user can check if their configuration at least typecheck.

If anything you can provide a way to check the configuration file (much like nginx -t does) so that you can merge the defaults in the application and allow the user to truly omit the optional fields

This problem is solved by the packaging and delivery of the application, many applications and services have defaults files editable or inspectable by the user…

#13

Ah yes, I see now why my suggestion doesn’t work.

One solution here might be forward declarations and a custom primitive.

Anyway, this seems like an important issue to solve.

#14

@srid: There is another solution I haven’t seen mentioned yet, which is to change the Haskell dhall package to be more tolerant of missing fields when marshaling into Haskell (e.g. treat a missing field as None). I haven’t really thought through the implications of that, but I wanted to throw out that idea for discussion.

A related solution along the same lines which doesn’t require changes to the dhall package is for your application to attempt to decode older schema versions if the configuration doesn’t match the latest schema version.

1 Like
#15

@zarel The idea is that one would run the neuron command on a directory like this: https://github.com/srid/neuron/tree/master/guide which contains a neuron.dhall file, and that would be enough. So it’s actually important that the neuron.dhall file is independent of the version of neuron being run, so that a user directory can be compatible with different versions.

#16

@Gabriel439 After some thought I like this idea. The reasoning is that this is what I would expect from reading some JSON: if a field is missing and is meant to be deserialized into a Maybe Something, then I expect to get Nothing rather than an error. This is what dhall-rust does, because I want it to be as much as possible a drop-in replacement for json and yaml.

I thought initially that this went against the strong typedness of dhall, but I now think those are two separate mechanisms: when deserializing something, you may ask dhall to check that it matches a given type, but the conversion to your Haskell value can be an independent process. This allows e.g. deserializing a dhall record into a HashMap even though the hashmap cannot be given a sensible dhall record type in general.

#17

Ah, I just thought of another solution we didn’t consider: instead of reading user.dhall directly in your application, you instead read a custom file containing:

{ author = None Text
, siteBaseUrl = None Text
, editUrl = None Text } // ./user.dhall

Since this file is under your control, you can add fields without the user needing to change anything. If the configuration type becomes more complicated, you can implement more complicated merging logic here too.

2 Likes
#18

@Nadrieril: I like that idea even better

#19

I tried playing with this approach of defining a builtin in Haskell, but for some reason Dhall is not recognizing it as record (even though it is?):

image
image1166Ă—1654 125 KB

#20

Yea, the // operator seems like a neat trick. At first pass, it seems to work - let me play with it further.

EDIT: This solved it, and I’m happy to stay with dhall. :slight_smile:

1 Like