Text manipulation functions


#41

As a big fan of dhall, I just wanted to chime in in favour of text introspection as an escape hatch (and not just at the import level). I’m trying to evangelize dhall for some promising use cases at work. For the most part, dhall is a great fit, I can imagine it working quite well. But I have a growing belief that some small parts will require text introspection. It’s a long story, but in tl;dr it’s likely one part will need to manipulate kubernetes manifests, so that’d be a choice between “use text manipulation” and “rewrite the entire dhall-kubernetes type tree with custom unions in most places where there are strings”.

So, I’m a devoted user who’s motivated to do things the right way, because I understand the benefit. But even I’m not sure I can wholeheartedly recommend dhall for this before knowing the entire solution, since there’s no escape hatch.


#42

@timbertson: For your use case, do you mainly need introspection in order to convert a Text field into an enum or are there other Text introspection facilities that you require?

The reason I ask is that we could add support for a toUnion keyword that, given a enum and an “immediate” (i.e. non-abstract) Text value, converts that Text value to an alternative of the same name, and fails at type-checking time otherwise (if the Text literal does not match any alternative name). This would be in roughly the same spirit as the fromMap keyword which was discussed here:

To answer your broader question, we’ll generally do our best to work with you to find a solution that balances user ergonomics with language security guarantees.

The user experience I would like to preserve is that once import resolution and type-checking succeed the user can then have supreme confidence that the configuration will “just work”. This is why I keep brainstorming ways to move Text introspection checks into either the import resolution phase or the the type-checking phase.


#43

I don’t think that would help, if I understand you correctly. The problem is we would be doing this on Text values passed in (as part of a kubernetes manifest), so it wouldn’t be a Text literal.

Simple example: imagine writing a function which adds a given annotation to a kubernetes deployment, but only if it doesn’t already exist. You’re given an instance of the Deployment type, and it’s impractical to modify that type so that all annotations are represented as a union (since that would mean forking the whole type hierarchy of dhall-kubernetes).


If the operation couldn’t fail, then could it be relaxed to all Text values, not just literals? What if toUnion always required a fallback case?

e.g. toUnion <OptionA | OptionB | Unknown: Text>

That would ensure everything’s complete, and puts the burden of “what to do when it isn’t a known value” back on the user.

(It should probably be an Either type though, rather than embedding an Unknown branch in each enum)


#44

I’ve always thought that Text being opaque was a core feature of Dhall.

That said, if we do add some Text operations, we should be sure to not offer any that use the accursed “codepoints” in any way. Codepoints barely deserve to exist, and shouldn’t be exposed in anything as high-level as Dhall.


#45

I guess I don’t know what you mean exactly by “just work”. Presumably any text manipulation would be total (i.e. returning Optional instead of failing, etc)? So the overall expression would be just as safe / correct, you’d just be allowing dramatically more branching logic.

For the contexts I’m using dhall, the end consumer of this config (kubernetes, github actions, etc) are already great big piles of runtime checks, so pushing any of that logic into dhall (where at least it’s pure) is improving the “just works” likelihood.


#46

@timbertson: I’m using the phrase “just work” as a shorthand for restating the idea that errors are caught at import resolution and type-checking time.

There are some built-ins within the language that return an Optional value (e.g. List/head) so we’re not strongly opposed to using Optional for programming logic. That said, enabling Text introspection, even total Text introspection, is different because it is prone to users creating weakly-typed DSLs embedded within Dhall (see my comment about comma-separated strings).

I understand that there are external tools and formats that might have some sort of weakly-typed structure, which is why we strive to provide adapter tools that convert from weakly-typed representations to Dhall (e.g. yaml-to-dhall).

That brings me to my next train of thought: I would like to lean a bit more on getting yaml-to-dhall to work for your use case. If I understood your original comment: the issue is that the default dhall-kubernetes schemas use Text in a few places where you would prefer a more precise union type. Would that issue be resolved if there were an easy way to update the deeply nested types?

In other words, what if there were a type-level analog of the current with keyword, so that you could do something like this:

let UpdateType = < RollingUpdate | OnDelete >

let MyDeployment = k8s.Deployment.Type with spec.?.strategy.?.type = UpdateType

… then you could use that customized type for your yaml-to-dhall conversion.


#47

Hmm, it’s hard to be sure. First thing to clarify is that it probably woudn’t involve yaml-to-dhall, the intent is for everyone to be writing dhall. Which means it’s (relatively) easy - people would be writing the union directly, rather than trying to get it through yaml-to-dhall.

That construct feels useful in general (so if it’s something you’re considering anyway you have my vote), but then you’ve still got the problem that any function built to work with the vanilla dhall-kubernetes types wouldn’t work on your modified type. And in particular it feels like it might be very difficult to get an ergonomic “clone” of the dhall-kubernetes package, in particular having all the various default records target the modified type. I haven’t looked too deep into it, but it feels like you could run into problems other than just the raw types.