Writing & validating appbuilder configuration with CUE

[jpluscplusm]

When I find an awesome tool that asks me to configure it with YaML, I try and figure out how I can actually configure it using CUE :-) CUE lets me write, validate, abstract and understand my configs really nicely.

I'm going to put a series of replies in this discussion that show the evolution of using CUE to write and validate appbuilder configs: starting with simply importing the currently-defined schema from the upstream repo into CUE's syntax and validating appbuilder configs; moving into writing appbuilder configs directly in CUE; and pointing forwards to writing opinionated and constraining abstractions over the top of appbuilder's generality using CUE.

Hopefully this discussion thread is useful for folks rolling the tool out into their orgs who want to help their users Do The Right Thing, and Not Shoot Themselves In The Foot. If it also happens to silently make a case for adopting CUE in other places then that's entirely unintentional; and also fine :-)

[jpluscplusm]

1: Importing & using upstream's schema in CUE

Here, we'll grab the upstream schema along with 2 examples of appbuilder configs, and show a few things:

1. the CUE schema definition, even the auto-generated, untouched-by-human-optimisation version, is more compact and readable than the JSON Schema it's derived from
2. the import is totally hands-free
3. it correctly catches intentionally invalid config
4. it highlights some perhaps-interesting nuances in the upstream schema

Note that whilst CUE is a Go-lang library, aimed at projects validating their users' inputs and configs, it's also a CLI tool - which is what we'll be using primarily.

---

For reproducibility, the process is codified in the following Makefile.

Click to expand, mouse-over the content to pop up a "copy" button in the top right

SRC_SCHEMA_URL=https://raw.githubusercontent.com/choria-io/go-choria/19ffe32b817c2746aa978cba81ebb70c690910b3/internal/fs/schemas/builder.json
SRC_APP_NATSCTL_URL=https://gist.githubusercontent.com/ripienaar/44fb9516ffb736bd9d5d2b280c533819/raw/75dde5827dfc256e930ceaca6ee10580dd9f39ef/gistfile1.txt
SRC_APP_DEMO_IN_README_URL=https://raw.githubusercontent.com/choria-io/appbuilder/cd7e8e940965a72bcdd92bf38f7ad51219cf3f55/README.md

clean: 
	rm -f builder.json natsctl.yaml builder.cue demo.yaml INVALID.demo.yaml
builder.json:
	curl -s --location $(SRC_SCHEMA_URL) >builder.json
natsctl.yaml:
	curl -s --location $(SRC_APP_NATSCTL_URL) >natsctl.yaml
demo.yaml:
	curl -s --location $(SRC_APP_DEMO_IN_README_URL) | sed -n '40,68p' > demo.yaml

builder.cue: builder.json
	cue import jsonschema: builder.json -f

.PHONY: validate-sources
validate-sources: natsctl.yaml demo.yaml builder.cue
	cue vet -c builder.cue natsctl.yaml
	cue vet -c builder.cue demo.yaml

INVALID.demo.yaml: demo.yaml
	sed '13s/exec/Exec/' demo.yaml > INVALID.demo.yaml

.PHONY: validate-invalid-demo
validate-invalid-demo: INVALID.demo.yaml builder.cue
	cue vet -c builder.cue INVALID.demo.yaml

---

To use the Makefile, we'll need a standard *nix toolset, along with the CUE cli, which is a single binary download. Note that the Makefile pins its inputs at specific commits - this is purely for reproducibility's sake, and has nothing to do with the underlying import or code.

First, let's generate the CUE version of the upstream schema:

$ make builder.cue 
curl -s --location https://raw.githubusercontent.com/choria-io/go-choria/19ffe32b817c2746aa978cba81ebb70c690910b3/internal/fs/schemas/builder.json >builder.json
cue import jsonschema: builder.json -f

---

Here's the contents of the auto-generated `builder.cue`:

import "strings"

// io.choria.builder.v1.application
//
// Choria Builder Application Specification
@jsonschema(schema="http://json-schema.org/draft-07/schema")

// A unique name for this application
name:        #shortname
description: #description
version:     #semver

// Contact details for the author
author: strings.MinRunes(1)

// A list of commands that make up the application
commands: #commands & [_, ...]

#shortname: strings.MinRunes(1) & =~"^[a-z0-9_-]*$"

#semver: strings.MinRunes(5) & =~"^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)(?:-((?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\\.(?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\\+([0-9a-zA-Z-]+(?:\\.[0-9a-zA-Z-]+)*))?$"

#rpc_filter: {
	// Target nodes in a specific sub-collective, defaults to
	// configured main collective
	collective?: string

	// List of fact filters, like operatingsystem=CentOS, as typed in
	// -F filters on the CLI
	facts?: [...string]

	// List of agents to require, as typed in -A filters on the CLI
	agents?: [...string]

	// List of classes to match on, as typed in -C filters on the CLI
	classes?: [...string]

	// List of identities to target, as typed in -I filters on the CLI
	identities?: [...string]

	// List of combined filters to match on targets, as typed in -W
	// filters on the CLI
	combined?: [...string]

	// Compound filter to use, as typed in -S filters on the CLI
	compound?: string

	// The discovery method to use
	discovery_method?: "mc" | "broadcast" | "choria" | "puppetdb" | "external" | "flatfile" | "file" | "inventory"

	// Number of seconds to allow for discovery to complete
	discovery_timeout?: int & >=0

	// Enables windowed dynamic discovery timeout
	dynamic_discovery_timeout?: bool

	// Path to a file listing node names in text or JSON format
	nodes_file?: string

	// Discovery options as a map of string values
	discovery_options?: {
		...
	}
	...
}

#standard_command: {
	// A unique name for this command
	name:        #shortname
	description: #description

	// When set always confirms the operation using the value as
	// prompt
	confirm_prompt?: string
	type:            _
	...
}

#description: strings.MinRunes(1)

#generic_flag: {
	// A unique name for this flag
	name:        #shortname
	description: #description

	// Indicates this flag must be passed
	required?: bool | *false

	// String to show as value place holder in help output
	placeholder?: string
	...
}

#generic_transform: {
	// A JQ query to pass the data through
	query: string
	...
}

#generic_argument: {
	// A unique name for this argument
	name:        #shortname
	description: #description

	// Indicates that this flag must be passed
	required?: bool | *false
	...
}

#commands: [...(#rpc_command | #parent_command | #kv_command | #exec_command | #discover_command) & _]

#parent_command: #standard_command & {
	type: "parent"

	// Additional CLI commands to add
	commands?: #commands & [_, ...]
	...
}

#discover_command: #standard_command & {
	type: "discover"

	// Optional filters to apply to the request, will be merged with
	// standard options if set
	filter?: #rpc_filter

	// Enables standard RPC filters like -C, -I etc
	std_filters?: bool | *false

	// List or arguments to accept after the command name
	arguments?: [...#generic_argument]

	// List of flags to add to the command
	flags?: [...#generic_flag]
	...
}

#exec_command: #standard_command & {
	type: "exec"

	// Additional CLI commands to add
	commands?: #commands

	// The command to execute, supports template interpolation
	command: strings.MinRunes(1)
	...
}

#kv_command: #standard_command & {
	type: "kv"

	// Additional CLI commands to add
	commands?: #commands

	// The name of the Key-Value store bucket
	bucket: =~"\\A[a-zA-Z0-9_-]+\\z" & strings.MinRunes(1)

	// The key to act on
	key: strings.MinRunes(1)

	// The value to store for the put operation
	value?: string

	// The action to perform against the bucket and key
	action: "get" | "put" | "del" | "history"

	// Renders the result in JSON format for get and history actions
	json?: bool | *false
	...
}

#rpc_command: #standard_command & {
	type: "rpc"

	// Additional CLI commands to add
	commands?:  #commands
	transform?: #generic_transform

	// Details of the RPC request
	request?: {
		// The agent to call
		agent?: #shortname

		// The action to call
		action?: #shortname

		// Free form list of input arguments as a hash, values support
		// template interpolation
		inputs?: {
			[string]: string
		}

		// Optional filters to apply to the request, will be merged with
		// standard options if set
		filter?: #rpc_filter
		...
	}

	// Enables standard RPC filters like -C, -I etc
	std_filters?: bool | *false

	// Enable flags to adjust the output format like --json, --table
	// etc
	output_format_flags?: bool | *false

	// Sets a specific output format, not compatible with
	// output_format_flags
	output_format?: "senders" | "json" | "table"

	// Enables the --display flag, not compatible with display
	display_flag?: bool | *false

	// Force a specific display format to be used, not compatible with
	// display_flags
	display?: "ok" | "failed" | "all" | "none"

	// Disables the progress bar
	no_progress?: bool

	// Batch size to use when executing the RPC request, not
	// compatible with batch_flags
	batch?: int

	// When batch is given this is the seconds to sleep between
	// batches, not compatible with batch_flags
	batch_sleep?: int

	// Enables the --batch and --batch-sleep flags
	batch_flags?: bool | *false

	// Prompts for confirmation when no filter is active resulting on
	// an action against all nodes
	all_nodes_confirm_prompt?: string

	// List of flags to add to the command
	flags?: [...#generic_flag & {
		// Choria reply filter
		reply_filter?: string
		...
	}]
	...
}
...

---

Now, let's use this file to validate 2 known-good examples of appbuilder config:

$ make validate-sources 
curl -s --location https://gist.githubusercontent.com/ripienaar/44fb9516ffb736bd9d5d2b280c533819/raw/75dde5827dfc256e930ceaca6ee10580dd9f39ef/gistfile1.txt >natsctl.yaml
curl -s --location https://raw.githubusercontent.com/choria-io/appbuilder/cd7e8e940965a72bcdd92bf38f7ad51219cf3f55/README.md | sed -n '40,68p' > demo.yaml
cue vet -c builder.cue natsctl.yaml
cue vet -c builder.cue demo.yaml
$ echo $?
0

The cue vet command shows us that when we unify the contents of all the file arguments we give the CLI, they don't disagree with each other. This is one way of using the cue CLI but, as we'll see in later replies, it's far from the best or most idiomatic way to validate structured data with CUE.

Now let's create an intentionally-invalid config, by subtly changing the demo config (it's the demo config extracted from the README in the appbuilder repo) to reference an "Exec" command type instead of the correct "exec" casing:

$ make validate-invalid-demo 
sed '13s/exec/Exec/' demo.yaml > INVALID.demo.yaml
cue vet -c builder.cue INVALID.demo.yaml
commands.0: 4 errors in empty disjunction:
commands.0.type: conflicting values "discover" and "parent":
    ./INVALID.demo.yaml:9:14
    ./builder.cue:17:11
    ./builder.cue:110:13
    ./builder.cue:110:80
    ./builder.cue:121:8
commands.0.type: conflicting values "exec" and "parent":
    ./INVALID.demo.yaml:9:14
    ./builder.cue:17:11
    ./builder.cue:110:13
    ./builder.cue:110:64
    ./builder.cue:139:8
commands.0.type: conflicting values "kv" and "parent":
    ./INVALID.demo.yaml:9:14
    ./builder.cue:17:11
    ./builder.cue:110:13
    ./builder.cue:110:50
    ./builder.cue:150:8
commands.0.type: conflicting values "rpc" and "parent":
    ./INVALID.demo.yaml:9:14
    ./builder.cue:17:11
    ./builder.cue:110:13
    ./builder.cue:110:17
    ./builder.cue:173:8
make: *** [Makefile:27: validate-invalid-demo] Error 1

Well, whilst it's definitely blowing up un-avoidably and noisily, such that we wouldn't try and use the INVALID.demo.yaml file, it's not showing the most helpful error messages! This is partly due to our use of the auto-generated schema, and partly because CUE's error messages aren't yet optimised for humans to read :-) The upstream project tracks this UX deficiency here, and a significant consumer of CUE (Dagger) also tracks it here. We'll also make some improvements in this area as this discussion thread expands over time.

Looking back at the builder.cue contents, check inside the #standard_command object. CUE has interpreted the JSON Schema definition as restricting the type key to _ - which is CUE's wildcard value: absolutely anything is valid, but it must exist. This points to some potential ambiguity, or could-be-tightened-up-ness, in the upstream schema. Despite being only a very slight smell, here in CUE, it's quite visible and stands out as A Bit Odd. Contrast with the upstream JSON Schema, where whatever has caused this looks relatively standardised and indistinguishable from "JSON Schema As Usual"

To finish, let's compare the volume of code. Now, whilst it's possible to represent any JSON Schema as a single, long line of JSON, the canonical form of the schema is more verbose than the CUE equivalent ... by quite some margin:

$ jq < builder.json | wc -l
538
$ cue fmt builder.cue
$ cat builder.cue | wc -l
243
$ # *including* how many blank lines of human-friendly white-space? :-)
$ grep -c ^$ builder.cue 
62

The next replies in this discussion will improve this auto-generated schema. We'll also start to tease apart the difference between a policy-based schema and a human-delighting abstraction - because those 2 roles aren't the same!

[ripienaar]

This looks pretty great, but I been holding off replying until the "next replies in this discussion" just fyi, not forgot about this :)

At the moment I steped away a bit from the schema based validation as in the end the validators just isnt good enough to handle many possible permutations and to figure out intent correctly. So now just validating in code. So very much still looking for good options here