Announcing Pydhall, another Python implementation of Dhall

#1

I started Pydhall as an educational project to explore both Dhall and functional programing paradigms in depth. It got a bit out of control and it’s now very close to be a working 16.0.0 compliant implementation, so I decided to make this formal annoucement.

Here’s a summary of the README:

What works

Since yesterday Pydhall can compute an erroneous hash of the Prelude (well… you have to comment out an assert in Text/show for this to happen). It correctly computes the hash of all modules except for JSON and Natural.

  • parsing
  • envvar and local files imports
  • typechecking
  • αβ-normalization
  • binary-encoding
  • semantic hash

Most of the spec tests pass: 62 failed, 1249 passed, 77 xfailed failures are in the not implemented binary-decode tests.

What doesn’t work yet

  • HTTP imports
  • binary-decoding

What’s not even clearly designed:

At the moment there is no clear python API to use the generated data structures. A few of them implement an as_python() method. I think we need a pythonic way to describe a dhall schema.

Thoughts / AMA prompts

The technical details and features are described in the readme. This post is a better place to discuss the Dhall ecosystem implications.

dhall-python

Of course, I’m aware of @SupraSummus work on a python implementation and I acknowledge that pydhall splits the effort, that’s why I was reluctant to talk about the project, before. I’m open to discuss the technically comparison between the projects.

The second hard question is what if pydhall becomes an official implementation ? I’m new to Dhall and I don’t even use it professionaly. I have the feeling that @SupraSummus is both more legitimate and insightful than I am to give their voice in the specification process, even if the dhall-python project is somewhat stale. I guess this two-implementations-for-one-language type of situation is new, and we will have to sort it out when/if Pydhall claims the official implementation status.

dhall-golang

Pydhall is a port of dhall-golang. It’s not clear at a first glance, because Pydhall uses the OOP/Python style (methods on terms and values) whereas dhall-golang uses the procedural/golang style (big fat typeswitches). Also, Pydhall has a PEG definition that is heavily inspired by (read copy-pasted from) dhall-golang’s.

This decision is based on the fact that I can read Go more fluently than any other language implementing Dhall. I believe it’s the case for most python developer potentially interested in contributing to Pydhall. Also, the process of porting from golang to python is pretty straightforward.

It was a good decision that eased the bootstrapping of the project a lot. I have no idea if Pydhall will stick to dhall-golang’s design, but I hope that when Pydhall catch up with the standard, the two projects could fuel each other if they don’t diverge dramatically in design. I’ve already spotted and reported bugs in dhall-golang. More eyeballs, less bugs :).

Future

When all the spec tests pass we have to design the python API, for injecting dhall computed values into the python code, defining Dhall types in python, at higher level than the internal type Values and defining custom builtins. I project to battle-test the design by implementing a dhall front-head to either docker-compose or ansible (maybe re-using @tristanC’s work).

I’m a dhall implementer, now. AMA.

7 Likes
#2

Also this post is a call for contributions :). I didn’t make this announcement earlier because the code was subjects to heavy refactoring. It’s stable enough now, and I’m more than open to contributions.

#3

Hooray for more implementations! :tada: Great work getting this far.

This question already has an answer. If you look in dhall-lang/.github/CONTRIBUTING.md under “How do changes get approved?” there are a few relevant things to note: first, dhall-python is not currently an official client (so there is no current official client for python), and second, this important quote:

Implementations do not need to be useful or widely used to get a vote. If you create a complete implementation as a side project that nobody uses and does not integrate with anything, you still get to vote on changes to the standard.

Furthermore, I’d add that if Pydhall is more up-to-date standardwise than dhall-python, it’s probably closer to being a useful implementation?

In short: I would have no problem supporting a proposal to make this implementation official, at such time as you feel it is “ready” (which is pretty much your own personal judgment).

Note also there is no pressure to make your implementation official if this is not what you want.

2 Likes
#4

Well, you clearly played a great part in this achievement. The go code made formal specifications a lot more clear to me, and I blindly followed your resolve-typecheck-eval-quote design, without having to re-inventing it.
( fun fact : I deleted yesterday a comment stating # TODO: understand and document this. The dumb, clueless, port from dhall-golang was perfectly working at that time.)

I thought it was, as IIRC it’s linked from the implementations section of the website. EDIT: It’s not

That’s the hard part. At the moment it’s a toy project, I made for fun. It’s also a lot of work and way less fun to maintain it. If I can’t use it professionally, when all is done and working, I’ll have to find the motivation to work on this. Owning an official implementation requires a bit of commitment I’m not sure I can provide.

OTOH, a python implementation is a much-needed one. If I manage build a small community of users around the project, its chances of survival would greatly improve (with or without myself involved). That’s the point of implementing Dhall front-ends to well-known, configuration-centred python application such as docker-compose and ansible. Anyone who has ever had to deal with a large, multi-repo, ansible configuration and knows Dhall can feel it.

#5

Owning an official implementation requires a bit of commitment I’m not sure I can provide.

This is one of the reasons why the voting instructions are carefully worded to say that implementations get a vote on the standard rather than individuals. The voting instructions do record a snapshot of the lead developers for each implementation (which can be updated), but it’s ultimately up to each implementation how they want to govern themselves and vote on standard changes.

This means that you would have the option to delegate responsibility to people more interested in maintaining the Python implementation and/or voting on standard changes.

As a concrete example, @sjakobi is also an active maintainer of the Haskell implementation, too, so even though I’m recorded as the lead developer for the Haskell implementation I will abstain from voting if we disagree on a certain proposal since the vote is supposed to represent the preferences of the Haskell maintainers, not just my preference. Similarly, if I go on vacation I leave him in charge of voting on behalf of the Haskell implementation.

#6

This design is called “normalization by evaluation” or NbE and has been discussed before. See this and following comments for some tutorials: Any notes on normalization by evaluation for dhall-haskell?

#7

Yes, I was aware of this, I mentioned NbE in the README of the project, yet. What I meant is that NbE is conceptual algorithm illustrated by code examples in the literature. A working implementation for Dhall, especially in a language I’m comfortable with was much easier to, read, to follow and to comprehend. It saved a lot of time and efforts.

Without the time and effort you put into dhall-golang I wouldn’t be any near of a working implementation today. Thank you for that :).

1 Like