From 4dbf0f3f09d31fcca5094cf4131006ac01fdbdde Mon Sep 17 00:00:00 2001 From: ShahanaFarooqui Date: Thu, 18 Jan 2024 11:07:45 -0800 Subject: [PATCH] doc: Schema generation instructions update This PR will generate doc/lightning-*.md file contents for rpc commands. It will get the information from consolidated doc/schemas/lightning-*.json and use it to generate markdown for RPC commands. Changelog-None. --- .../code-generation.md | 18 +----- .../writing-json-schemas.md | 30 ++++------ doc/schemas/WRITING_SCHEMAS.md | 55 ++++++++----------- 3 files changed, 36 insertions(+), 67 deletions(-) diff --git a/doc/contribute-to-core-lightning/code-generation.md b/doc/contribute-to-core-lightning/code-generation.md index 5eba77849750..2af618109396 100644 --- a/doc/contribute-to-core-lightning/code-generation.md +++ b/doc/contribute-to-core-lightning/code-generation.md @@ -3,7 +3,7 @@ title: "Code Generation" slug: "code-generation" hidden: false createdAt: "2023-04-22T12:29:01.116Z" -updatedAt: "2023-04-22T12:44:47.814Z" +updatedAt: "2024-01-18T12:44:47.814Z" --- The CLN project has a multitude of interfaces, most of which are generated from an abstract schema: @@ -13,19 +13,11 @@ The CLN project has a multitude of interfaces, most of which are generated from 1. addition of FD passing semantics to allow establishing a new connection between daemons (communication uses [socketpair](https://man7.org/linux/man-pages/man2/socketpair.2.html), so no `connect`) 2. change the message length prefix from `u16` to `u32`, allowing for messages larger than 65Kb. The CSV files are with the respective sub-daemon and also use [generate-wire.py](https://github.com/ElementsProject/lightning/blob/master/tools/generate-wire.py) to generate encoding, decoding and printing functions -- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `.request.json` for the request message, and a `.schema.json` for the response (the mismatch is historical and will eventually be addressed). During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`][msggen] tool. More on the generated stubs and utilities below. +- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `lightning-*.json` for request and response. During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`[msggen] tool. More on the generated stubs and utilities below. ## Man pages -The manpages are partially generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request schema and fills in the manpage between two markers: - -```markdown -[comment]: # (GENERATE-FROM-SCHEMA-START) -... -[comment]: # (GENERATE-FROM-SCHEMA-END) -``` - - +The manpages are generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request and response schema from `lightning-*.json` and generates markdown contents and manpages: > 📘 > @@ -51,10 +43,6 @@ The manpages are partially generated from the JSON schemas using the [`fromschem } [/block] - - - - ### `cln-rpc` We use `msggen` to generate the Rust bindings crate [`cln-rpc`](https://github.com/ElementsProject/lightning/tree/master/cln-rpc). These bindings contain the stubs for the JSON-RPC methods, as well as types for the request and response structs. The [generator code](https://github.com/ElementsProject/lightning/blob/master/contrib/msggen/msggen/gen/rust.py) maps each abstract JSON-RPC type to a Rust type, minimizing size (e.g., binary data is hex-decoded). diff --git a/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md b/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md index c2d4609275d3..4ee93743fedb 100644 --- a/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md +++ b/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md @@ -3,36 +3,24 @@ title: "Writing JSON Schemas" slug: "writing-json-schemas" hidden: false createdAt: "2023-01-25T05:46:43.718Z" -updatedAt: "2023-01-30T15:36:28.523Z" +updatedAt: "2024-01-18T15:36:28.523Z" --- -A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command responses, and also use it to generate our documentation. +A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command requests and responses, and also use it to generate our documentation. Yes, schemas are horrible to write, but they're damn useful. We can only use a subset of the full [JSON Schema Specification](https://json-schema.org/), but if you find that limiting it's probably a sign that you should simplify your JSON output. ## Updating a Schema -If you add a field, you should add it to the schema, and you must add "added": "VERSION" (where VERSION is the next release version!). +If you add a field, you should add it to the field schema, and you must add "added": "VERSION" (where VERSION is the next release version!). -Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version). They will be removed two versions later. +Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version) to the field. They will be removed two versions later. ## How to Write a Schema -Name the schema doc/schemas/`command`.schema.json: the testsuite should pick it up and check all invocations of that command against it. +Name the schema doc/schemas/lightning-`command`.json: the testsuite should pick it up and check all invocations of that command against it. +The core lightning RPC commands use custom schema specification defined in [rpc-schema-draft](https://github.com/ElementsProject/lightning/doc/rpc-schema-draft.json). -I recommend copying an existing one to start. - -You will need to put the magic lines in the manual page so `make doc-all` will fill it in for you: - -```json -[comment]: # (GENERATE-FROM-SCHEMA-START) -[comment]: # (GENERATE-FROM-SCHEMA-END) -``` - - - -If something goes wrong, try tools/fromscheme.py doc/schemas/`command`.schema.json to see how far it got before it died. - -You should always use `"additionalProperties": false`, otherwise your schema might not be covering everything. Deprecated fields simply have `"deprecated": true` in their properties, so they are allowed by omitted from the documentation. +I recommend copying an existing one to start. If something goes wrong, try tools/fromscheme.py doc/schemas/lightning-`command`.json to see how far it got before it died. You should always list all fields which are _always_ present in `"required"`. @@ -57,6 +45,10 @@ To add conditional fields: 6. Inside the "then", use `"additionalProperties": false` and place empty `{}` for all the other possible properties. 7. If you haven't covered all the possibilties with `if` statements, add an `else` with `"additionalProperties": false` which simply mentions every allowable property. This ensures that the fields can _only_ be present when conditions are met. +### Exceptions in dynamic schema generation + +- If response (`RETURN VALUE`) should not be dynamically generated and you want it to be a custom text message instead. You can use `return_value_notes` to add custom text with empty `properties`. Examples: `setpsbtversion`, `commando`, `recover` commands. + ### JSON Drinking Game! 1. Sip whenever you have an additional comma at the end of a sequence. diff --git a/doc/schemas/WRITING_SCHEMAS.md b/doc/schemas/WRITING_SCHEMAS.md index aff29fcef60e..bcc070e6540c 100644 --- a/doc/schemas/WRITING_SCHEMAS.md +++ b/doc/schemas/WRITING_SCHEMAS.md @@ -2,7 +2,7 @@ A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they -match command responses, and also use it to generate our +match command requests and responses, and also use it to generate our documentation. Yes, schemas are horrible to write, but they're damn useful. We can only @@ -12,46 +12,31 @@ your JSON output. ## Updating a Schema -If you add a field, you should add it to the schema, and you must add -"added": "VERSION" (where VERSION is the next release version!). +If you add a field, you should add it to the field schema, and you must +add "added": "VERSION" (where VERSION is the next release version!). -Similarly, if you deprecate a field, add "deprecated": "VERSION" (where -VERSION is the next release version). They will be removed two versions -later. +Similarly, if you deprecate a field, add "deprecated": "VERSION" +(where VERSION is the next release version) to the field. They will be +removed two versions later. ## How to Write a Schema -Name the schema doc/schemas/`command`.schema.json: the testsuite should -pick it up and check all invocations of that command against it. +Name the schema doc/schemas/lightning-`command`.json: the testsuite +should pick it up and check all invocations of that command against it. +The core lightning RPC commands use custom schema specification defined +in [rpc-schema-draft](https://github.com/ElementsProject/lightning/doc/rpc-schema-draft.json). -I recommend copying an existing one to start. +I recommend copying an existing one to start. If something goes wrong, +try tools/fromscheme.py doc/schemas/lightning-`command`.json to see +how far it got before it died. -You will need to put the magic lines in the manual page so `make doc-all` -will fill it in for you: +You should always list all fields which are _always_ present in `"required"`. -``` -[comment]: # (GENERATE-FROM-SCHEMA-START) -[comment]: # (GENERATE-FROM-SCHEMA-END) -``` +We extend the basic types; see [fixtures.py](https://github.com/ElementsProject/lightning/tree/master/contrib/pyln-testing/pyln/testing/fixtures.py). -If something goes wrong, try tools/fromscheme.py -doc/schemas/`command`.schema.json to see how far it got before it died. - -You should always use `"additionalProperties": false`, otherwise -your schema might not be covering everything. Deprecated fields -simply have `"deprecated": true` in their properties, so they -are allowed by omitted from the documentation. - -You should always list all fields which are *always* present in -`"required"`. - -We extend the basic types; see [fixtures.py][fixtures]. - -[fixtures]: https://github.com/ElementsProject/lightning/blob/master/contrib/pyln-testing/pyln/testing/fixtures.py - -In addition, before committing a new schema or a new version of it, make sure that it -is well formatted. If you don't want do it by hand, use `make fmt-schema` that uses -jq under the hood. +In addition, before committing a new schema or a new version of it, +make sure that it is well formatted. If you don't want do it by hand, +use `make fmt-schema` that uses jq under the hood. ### Using Conditional Fields @@ -80,6 +65,10 @@ To add conditional fields: mentions every allowable property. This ensures that the fields can *only* be present when conditions are met. +### Exceptions in dynamic schema generation + +- If response (`RETURN VALUE`) should not be dynamically generated and you want it to be a custom text message instead. You can use `return_value_notes` to add custom text with empty `properties`. Examples: `setpsbtversion`, `commando`, `recover` commands. + ### JSON Drinking Game! 1. Sip whenever you have an additional comma at the end of a sequence.