Skip to content

Commit

Permalink
doc: Schema generation instructions update
Browse files Browse the repository at this point in the history 
This PR will generate doc/lightning-*.md file contents for rpc commands. It will get the information from consolidated doc/schemas/lightning-*.json and use it to generate markdown for RPC commands.

Changelog-None.
  • Loading branch information
@ShahanaFarooqui
ShahanaFarooqui committed Jan 20, 2024
1 parent 4014af5 commit 4dbf0f3
Show file tree
Hide file tree
Showing 3 changed files with 36 additions and 67 deletions.
18 changes: 3 additions & 15 deletions doc/contribute-to-core-lightning/code-generation.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "Code Generation"
slug: "code-generation"
hidden: false
createdAt: "2023-04-22T12:29:01.116Z"
updatedAt: "2023-04-22T12:44:47.814Z"
updatedAt: "2024-01-18T12:44:47.814Z"
---
The CLN project has a multitude of interfaces, most of which are generated from an abstract schema:

Expand All @@ -13,19 +13,11 @@ The CLN project has a multitude of interfaces, most of which are generated from
1. addition of FD passing semantics to allow establishing a new connection between daemons (communication uses [socketpair](https://man7.org/linux/man-pages/man2/socketpair.2.html), so no `connect`)
2. change the message length prefix from `u16` to `u32`, allowing for messages larger than 65Kb. The CSV files are with the respective sub-daemon and also use [generate-wire.py](https://github.com/ElementsProject/lightning/blob/master/tools/generate-wire.py) to generate encoding, decoding and printing functions

- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `.request.json` for the request message, and a `.schema.json` for the response (the mismatch is historical and will eventually be addressed). During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`][msggen] tool. More on the generated stubs and utilities below.
- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `lightning-*.json` for request and response. During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`[msggen] tool. More on the generated stubs and utilities below.

## Man pages

The manpages are partially generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request schema and fills in the manpage between two markers:

```markdown
[comment]: # (GENERATE-FROM-SCHEMA-START)
...
[comment]: # (GENERATE-FROM-SCHEMA-END)
```


The manpages are generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request and response schema from `lightning-*.json` and generates markdown contents and manpages:

> 📘
>
Expand All @@ -51,10 +43,6 @@ The manpages are partially generated from the JSON schemas using the [`fromschem
}
[/block]





### `cln-rpc`

We use `msggen` to generate the Rust bindings crate [`cln-rpc`](https://github.com/ElementsProject/lightning/tree/master/cln-rpc). These bindings contain the stubs for the JSON-RPC methods, as well as types for the request and response structs. The [generator code](https://github.com/ElementsProject/lightning/blob/master/contrib/msggen/msggen/gen/rust.py) maps each abstract JSON-RPC type to a Rust type, minimizing size (e.g., binary data is hex-decoded).
Expand Down
30 changes: 11 additions & 19 deletions doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,36 +3,24 @@ title: "Writing JSON Schemas"
slug: "writing-json-schemas"
hidden: false
createdAt: "2023-01-25T05:46:43.718Z"
updatedAt: "2023-01-30T15:36:28.523Z"
updatedAt: "2024-01-18T15:36:28.523Z"
---
A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command responses, and also use it to generate our documentation.
A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command requests and responses, and also use it to generate our documentation.

Yes, schemas are horrible to write, but they're damn useful. We can only use a subset of the full [JSON Schema Specification](https://json-schema.org/), but if you find that limiting it's probably a sign that you should simplify your JSON output.

## Updating a Schema

If you add a field, you should add it to the schema, and you must add "added": "VERSION" (where VERSION is the next release version!).
If you add a field, you should add it to the field schema, and you must add "added": "VERSION" (where VERSION is the next release version!).

Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version). They will be removed two versions later.
Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version) to the field. They will be removed two versions later.

## How to Write a Schema

Name the schema doc/schemas/`command`.schema.json: the testsuite should pick it up and check all invocations of that command against it.
Name the schema doc/schemas/lightning-`command`.json: the testsuite should pick it up and check all invocations of that command against it.
The core lightning RPC commands use custom schema specification defined in [rpc-schema-draft](https://github.com/ElementsProject/lightning/doc/rpc-schema-draft.json).

I recommend copying an existing one to start.

You will need to put the magic lines in the manual page so `make doc-all` will fill it in for you:

```json
[comment]: # (GENERATE-FROM-SCHEMA-START)
[comment]: # (GENERATE-FROM-SCHEMA-END)
```



If something goes wrong, try tools/fromscheme.py doc/schemas/`command`.schema.json to see how far it got before it died.

You should always use `"additionalProperties": false`, otherwise your schema might not be covering everything. Deprecated fields simply have `"deprecated": true` in their properties, so they are allowed by omitted from the documentation.
I recommend copying an existing one to start. If something goes wrong, try tools/fromscheme.py doc/schemas/lightning-`command`.json to see how far it got before it died.

You should always list all fields which are _always_ present in `"required"`.

Expand All @@ -57,6 +45,10 @@ To add conditional fields:
6. Inside the "then", use `"additionalProperties": false` and place empty `{}` for all the other possible properties.
7. If you haven't covered all the possibilties with `if` statements, add an `else` with `"additionalProperties": false` which simply mentions every allowable property. This ensures that the fields can _only_ be present when conditions are met.

### Exceptions in dynamic schema generation

- If response (`RETURN VALUE`) should not be dynamically generated and you want it to be a custom text message instead. You can use `return_value_notes` to add custom text with empty `properties`. Examples: `setpsbtversion`, `commando`, `recover` commands.

### JSON Drinking Game!

1. Sip whenever you have an additional comma at the end of a sequence.
Expand Down
55 changes: 22 additions & 33 deletions doc/schemas/WRITING_SCHEMAS.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

A JSON Schema is a JSON file which defines what a structure should
look like; in our case we use it in our testsuite to check that they
match command responses, and also use it to generate our
match command requests and responses, and also use it to generate our
documentation.

Yes, schemas are horrible to write, but they're damn useful. We can only
Expand All @@ -12,46 +12,31 @@ your JSON output.

## Updating a Schema

If you add a field, you should add it to the schema, and you must add
"added": "VERSION" (where VERSION is the next release version!).
If you add a field, you should add it to the field schema, and you must
add "added": "VERSION" (where VERSION is the next release version!).

Similarly, if you deprecate a field, add "deprecated": "VERSION" (where
VERSION is the next release version). They will be removed two versions
later.
Similarly, if you deprecate a field, add "deprecated": "VERSION"
(where VERSION is the next release version) to the field. They will be
removed two versions later.

## How to Write a Schema

Name the schema doc/schemas/`command`.schema.json: the testsuite should
pick it up and check all invocations of that command against it.
Name the schema doc/schemas/lightning-`command`.json: the testsuite
should pick it up and check all invocations of that command against it.
The core lightning RPC commands use custom schema specification defined
in [rpc-schema-draft](https://github.com/ElementsProject/lightning/doc/rpc-schema-draft.json).

I recommend copying an existing one to start.
I recommend copying an existing one to start. If something goes wrong,
try tools/fromscheme.py doc/schemas/lightning-`command`.json to see
how far it got before it died.

You will need to put the magic lines in the manual page so `make doc-all`
will fill it in for you:
You should always list all fields which are _always_ present in `"required"`.

```
[comment]: # (GENERATE-FROM-SCHEMA-START)
[comment]: # (GENERATE-FROM-SCHEMA-END)
```
We extend the basic types; see [fixtures.py](https://github.com/ElementsProject/lightning/tree/master/contrib/pyln-testing/pyln/testing/fixtures.py).

If something goes wrong, try tools/fromscheme.py
doc/schemas/`command`.schema.json to see how far it got before it died.

You should always use `"additionalProperties": false`, otherwise
your schema might not be covering everything. Deprecated fields
simply have `"deprecated": true` in their properties, so they
are allowed by omitted from the documentation.

You should always list all fields which are *always* present in
`"required"`.

We extend the basic types; see [fixtures.py][fixtures].

[fixtures]: https://github.com/ElementsProject/lightning/blob/master/contrib/pyln-testing/pyln/testing/fixtures.py

In addition, before committing a new schema or a new version of it, make sure that it
is well formatted. If you don't want do it by hand, use `make fmt-schema` that uses
jq under the hood.
In addition, before committing a new schema or a new version of it,
make sure that it is well formatted. If you don't want do it by hand,
use `make fmt-schema` that uses jq under the hood.

### Using Conditional Fields

Expand Down Expand Up @@ -80,6 +65,10 @@ To add conditional fields:
mentions every allowable property. This ensures that the fields
can *only* be present when conditions are met.

### Exceptions in dynamic schema generation

- If response (`RETURN VALUE`) should not be dynamically generated and you want it to be a custom text message instead. You can use `return_value_notes` to add custom text with empty `properties`. Examples: `setpsbtversion`, `commando`, `recover` commands.

### JSON Drinking Game!

1. Sip whenever you have an additional comma at the end of a sequence.
Expand Down

0 comments on commit 4dbf0f3

Please sign in to comment.