Text manipulation functions

#35

@Profpatsch: Please try to avoid phrasing things in a confrontational way. If you disagree, address the substance of what they are saying instead of appealing to ridicule.

Also, I do believe there is a sensible way to implement Text/equal. The document that @blamario linked to might appear imposing, but it essentially boils down to: two Text values are equal if their normal forms are equal, and normal forms are well-understood for Unicode (any mainstream language with a Unicode library will support this)

The reason we haven’t yet added Text/equal isn’t because of concerns over Unicode compliance, but rather due to trying to design the language to work without it as long as possible.

4 Likes
#36

For reference, here’s the Eq function for the built-in, guaranteed-to-be-utf8 str in rust

    /// Implements ordering of strings.
    ///
    /// Strings are ordered  lexicographically by their byte values. This orders Unicode code
    /// points based on their positions in the code charts. This is not necessarily the same as
    /// "alphabetical" order, which varies by language and locale. Sorting strings according to
    /// culturally-accepted standards requires locale-specific data that is outside the scope of
    /// the `str` type.
    #[stable(feature = "rust1", since = "1.0.0")]
    impl Ord for str {
        #[inline]
        fn cmp(&self, other: &str) -> Ordering {
            self.as_bytes().cmp(other.as_bytes())
        }
    }

    #[stable(feature = "rust1", since = "1.0.0")]
    impl PartialEq for str {
        #[inline]
        fn eq(&self, other: &str) -> bool {
            self.as_bytes() == other.as_bytes()
        }
        #[inline]
        fn ne(&self, other: &str) -> bool {
            !(*self).eq(other)
        }
    }

I don’t see the collation algorithms coming with the rust standard library.

Haskell’s Text type also does bytewise comparison as far as I can see.

From the collation standard:

Also, defaults in CLDR data may vary by locale. For example, normalization is turned off in most CLDR locales (those that don’t normally use multiple accents). The default for strength in UCA is tertiary ; it can be changed for different locales in CLDR.

Another point: the static dhall binary is about 3MB right now. How big are the collation data tables? I assume they will have to be embedded?

#37

Here’s a discussion about unicode normalization in the rust stdilb: https://internals.rust-lang.org/t/rust-unicode/2730/18

There was also a very good, pretty exhaustive blog post somewhere, but I can’t find it right now.

#38

Here’s a good comment from that thread:

As the original author of the NF(K){C,D} implementations that were originally in libunicode and are now in the unicode-normalization crate I feel obliged to chime in here.
I think performing normalization as part of PartialEq::eq to “fix” string equality would be misguided. Different applications want different normalization, and (as others have said) thinking normalization on its own is sufficient is naive in many cases.

#39

My answer was indeed somewhat tongue-in-cheek. Sorry I missed the ensuing discussion, which I am quite glad to see.

I actually agree with the Rust conclusion that Dhall’s default string equality should not include full Unicode normalization. The discussion you linked already brings up the complexity argument and the fact that many applications need exact (byte-wise or code-point-wise) equality. To me the crucial point is that Unicode defines the (already linked) canonical normalizations that together with the primitive exact equality give you a canonical equality. I’m not sure if the relationship between code-point-wise and lexicographical ordering is as neat, but we can cross that bridge when we get there.

My larger point is that people have already thought through these questions, and they have provided clear answers (down to implementation algorithms) where possible. Every programming language designed in this century should follow the Unicode recommendations. Yes they’re complex (because writing is complex) but that’s no reason to give up.

1 Like
#40

I created an issue to track the idea of parsing Text at import resolution time similar to what @blamario proposed: https://github.com/dhall-lang/dhall-lang/issues/989

I know it’s not exactly germane to the original request of truncating Text, which we may still address in a separate way (perhaps a new Text/truncate built-in), but I just wanted to mention it briefly here since the topic had come up yet again recently.

3 Likes
#41

As a big fan of dhall, I just wanted to chime in in favour of text introspection as an escape hatch (and not just at the import level). I’m trying to evangelize dhall for some promising use cases at work. For the most part, dhall is a great fit, I can imagine it working quite well. But I have a growing belief that some small parts will require text introspection. It’s a long story, but in tl;dr it’s likely one part will need to manipulate kubernetes manifests, so that’d be a choice between “use text manipulation” and “rewrite the entire dhall-kubernetes type tree with custom unions in most places where there are strings”.

So, I’m a devoted user who’s motivated to do things the right way, because I understand the benefit. But even I’m not sure I can wholeheartedly recommend dhall for this before knowing the entire solution, since there’s no escape hatch.

1 Like
#42

@timbertson: For your use case, do you mainly need introspection in order to convert a Text field into an enum or are there other Text introspection facilities that you require?

The reason I ask is that we could add support for a toUnion keyword that, given a enum and an “immediate” (i.e. non-abstract) Text value, converts that Text value to an alternative of the same name, and fails at type-checking time otherwise (if the Text literal does not match any alternative name). This would be in roughly the same spirit as the fromMap keyword which was discussed here:

To answer your broader question, we’ll generally do our best to work with you to find a solution that balances user ergonomics with language security guarantees.

The user experience I would like to preserve is that once import resolution and type-checking succeed the user can then have supreme confidence that the configuration will “just work”. This is why I keep brainstorming ways to move Text introspection checks into either the import resolution phase or the the type-checking phase.

1 Like
#43

I don’t think that would help, if I understand you correctly. The problem is we would be doing this on Text values passed in (as part of a kubernetes manifest), so it wouldn’t be a Text literal.

Simple example: imagine writing a function which adds a given annotation to a kubernetes deployment, but only if it doesn’t already exist. You’re given an instance of the Deployment type, and it’s impractical to modify that type so that all annotations are represented as a union (since that would mean forking the whole type hierarchy of dhall-kubernetes).

If the operation couldn’t fail, then could it be relaxed to all Text values, not just literals? What if toUnion always required a fallback case?

e.g. toUnion <OptionA | OptionB | Unknown: Text>

That would ensure everything’s complete, and puts the burden of “what to do when it isn’t a known value” back on the user.

(It should probably be an Either type though, rather than embedding an Unknown branch in each enum)

#44

I’ve always thought that Text being opaque was a core feature of Dhall.

That said, if we do add some Text operations, we should be sure to not offer any that use the accursed “codepoints” in any way. Codepoints barely deserve to exist, and shouldn’t be exposed in anything as high-level as Dhall.

1 Like
#45

I guess I don’t know what you mean exactly by “just work”. Presumably any text manipulation would be total (i.e. returning Optional instead of failing, etc)? So the overall expression would be just as safe / correct, you’d just be allowing dramatically more branching logic.

For the contexts I’m using dhall, the end consumer of this config (kubernetes, github actions, etc) are already great big piles of runtime checks, so pushing any of that logic into dhall (where at least it’s pure) is improving the “just works” likelihood.

#46

@timbertson: I’m using the phrase “just work” as a shorthand for restating the idea that errors are caught at import resolution and type-checking time.

There are some built-ins within the language that return an Optional value (e.g. List/head) so we’re not strongly opposed to using Optional for programming logic. That said, enabling Text introspection, even total Text introspection, is different because it is prone to users creating weakly-typed DSLs embedded within Dhall (see my comment about comma-separated strings).

I understand that there are external tools and formats that might have some sort of weakly-typed structure, which is why we strive to provide adapter tools that convert from weakly-typed representations to Dhall (e.g. yaml-to-dhall).

That brings me to my next train of thought: I would like to lean a bit more on getting yaml-to-dhall to work for your use case. If I understood your original comment: the issue is that the default dhall-kubernetes schemas use Text in a few places where you would prefer a more precise union type. Would that issue be resolved if there were an easy way to update the deeply nested types?

In other words, what if there were a type-level analog of the current with keyword, so that you could do something like this:

let UpdateType = < RollingUpdate | OnDelete >

let MyDeployment = k8s.Deployment.Type with spec.?.strategy.?.type = UpdateType

… then you could use that customized type for your yaml-to-dhall conversion.

1 Like
#47

Hmm, it’s hard to be sure. First thing to clarify is that it probably woudn’t involve yaml-to-dhall, the intent is for everyone to be writing dhall. Which means it’s (relatively) easy - people would be writing the union directly, rather than trying to get it through yaml-to-dhall.

That construct feels useful in general (so if it’s something you’re considering anyway you have my vote), but then you’ve still got the problem that any function built to work with the vanilla dhall-kubernetes types wouldn’t work on your modified type. And in particular it feels like it might be very difficult to get an ergonomic “clone” of the dhall-kubernetes package, in particular having all the various default records target the modified type. I haven’t looked too deep into it, but it feels like you could run into problems other than just the raw types.

#48

I am sorry if this is the wrong place to ask but is there a way in dhall to assert that one user inputs a value containing lowercase only ?
We have encountered such problem in k8s where namespaces have to be lowercase (and we had received a foreign user input with a capital (e.g: name = “Foo”)

Somehow I wish for a type Text that only allows lowercase.

Of course if a function Text/lowerCase : Text → Text exists I could just do the transformation before converting in yaml with dhall-to-yaml (but that is not as nice in this case)

#49

I just wanted to note here that Text/replace was standardized, so now the original request can be resolved using Text/replace "-" "_"

1 Like
#50

@Gabriel439 is there a way to enforce Text to be lowercase only (as it is the case for k8s namespace) using the new Text/replace ?

1 Like
#51

@PierreR: You can approximate lowercase behavior using something like Text/replace "A" 'a' (Text/replace "B" 'b' …), but in general lowercasing a string in a Unicode-aware way requires a special-purpose built-in because the real rules are complicated. See:

http://www.unicode.org/versions/Unicode13.0.0/ch05.pdf#G21180

3 Likes
#52

perhaps that can be a next couple builtins!

Uppercasing/lowercasing a fun corner of text :stuck_out_tongue:

1 Like
#53

Following up yet again to mention that the Prelude does in fact have those suggested utilities for uppercasing and lowercasing ASCII Text:

1 Like
#54

I would love for dhall to have a rich support for text validation and i immediately realize there are many facilities required for a expressive validation systems. In my view, the text support should be gradually added in several parts:

  1. Provide List UnicodeScalar Primitive (or something similar) and other common unicode/ascii utilities for advanced users and library writer. Note that UnicodeScalar is perferred over codepoint to avoid the issue of surrogate pair
  2. Add minimalism text support for ordinary user, including span, splitAt, stripPrefix and Natural/fromText, Float/fromText etc.

The design and implementation for the first two stage should be easy and already useful for many users. The following design and implementation would be complicated.

  1. Support check up/extract information against particular grammar (regular grammar, ABNF, PEG, Attribute Grammar…).

Obviously, more expressive grammar leads to more precise validation, but it also brings implementation complexity.
In my opinion, a good implementation of 3 requires Inductive and User-Defined data type. First, RG and CFG can be regarded as an instance of an inductive data type (e.g. Grammar NonTerminal, Grammar as a type constructor, data constructor take production rules). Secondly, a parsing tree for given grammar rules is naturally a family of mutual inductive data types for each nonterminal. It would not be very ergonomics to continue use current roundabout. Third, certain validation/deserialization schemes can be associated with the corresponding data type.

Note that allowing user have access to parsing tree further enhance the validation ability. For example, it may allow PEG grammar be checked even if only ABNF is supported; or force that the close and open tag pair of xml are of the same element name (which can’t be done in CFG).