Demo: Configure GitLab pipelines using Dhall


#1

GitLab’s CI/CD pipelines are usually configured using a domain-specific language embedded in YAML (.gitlab-ci.yml). The DSL is complex, untyped, evolving, and provides limited validation capabilities.

GitLab 12.9 introduced dynamic child pipelines which means the primary pipeline controlled by gitlab-ci.yml can dynamically create a generated-config.yml file to spawn a child pipeline.

Here’s a proof of concept demo that leverages the feature to dynamically generate pipeline configurations from Dhall. This solves the problem of trying to keep the Dhall source of truth in sync with the static YAML config. It adds ~15s of overhead to the start of CI, but it may be possible to implement an optimistic caching strategy.

The repo doesn’t do the difficult work of modeling the GitLab DSL using Dhall types. That’s quite involved; as I mentioned the DSL evolves. Error messages from GitLab’s official validation tool are often unclear and I’ve also had issues getting it to resolve remote includes. I’ve tried 3rd party CLI-based validation tools and typically I bump against DSL features they don’t support. For now my intention is to focus more on modeling the “good parts” of the DSL which have been stable for a while and rely Dhall’s composability to make up for the rest.


#2

Good news!

  1. Around when I started this Ben Gamari also made a package to implement Dhall types for GitLab CI. There are more types to implement, but it’s already usable. I’ve integrated it into my demo repo and it works well.
  2. Last year there was a GitLab proposal to support Dhall in addition to YAML. I commented there to let them know about my demo. Then they created an issue to implement Dhall support using the same mechanism I am and exposing it to users, likely as an official CI template.

#3

Dang, that is good news! Auspicious stuff :slight_smile:


#4

Yes, that’s great news. We use gitlab at work and it’s painful to define complex pipelines with the YAML DSL. Removing code repetition amongst repos involved hacky and rather fragile solutions. Dhall in Gitlab-CI is a great way to convince my co-workers to give Dhall a try :).