add yanglint feature #193
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| # Working with YANG [RFC7950](https://tools.ietf.org/rfc7950) | ||
|
|
||
| In the bad old days, YANG modules were hand-copied into markdown files, but | ||
| this became untenable as YANG started getting more usage. | ||
|
|
||
| There are 3 YANG features that will be picked up during make, and which | ||
| will have enhanced error checking when the VALIDATE_YANG environment | ||
| variable is set to 1 or when running `make yanglint`: | ||
|
|
||
| * `YANG-MODULE <ietf-example.yang>`: inserts the text of the .yang file | ||
|
There was a problem hiding this comment. The first level of bullets should be flush with the left column. There was a problem hiding this comment. It's not clear whether this is a feature of the markdown processor or whether |
||
| before building the markdown. Usually inside a diagram. Inserts | ||
| \<CODE BEGINS\> and \<CODE ENDS\> tags in accordance with | ||
| [Section 3.2 of RFC 8407](https://tools.ietf.org/html/rfc8407#section-3.2) | ||
|
|
||
| * when VALIDATE_YANG=1 or during yanglint, also performs a pyang lint check for IETF rules. | ||
|
|
||
| * `YANG-TREE <ietf-example.yang>`: inserts the tree diagram of the .yang | ||
| file (as generated by pyang) before building the markdown, in | ||
| accordance with | ||
| [Section 3.4 of RFC 8407](https://tools.ietf.org/html/rfc8407#section-3.4) | ||
|
|
||
| * `YANG-DATA <ietf-example.yang> <example-data.json>`: inserts the text | ||
| of the .json file before building the markdown. | ||
|
|
||
| * when VALIDATE_YANG=1 or when running `make yanglint`, also performs a yanglint check to ensure that | ||
| the example-data.json validates against the model in ietf-example.yang. | ||
| (yanglint is part of the [libyang](https://github.com/CESNET/libyang) | ||
| project) | ||
|
|
||
| Note that the lint checks occur only for YANG modules and examples that | ||
| are imported with these commands from files external to the markdown. | ||
| If you have YANG examples or modules embedded in the markdown, they are | ||
| not examined by this feature. | ||
|
|
||
| ## Motivation | ||
|
|
||
| It's useful to maintain separate YANG module files, because these can be | ||
| dropped directly into various tools that make use of them. By incorporating | ||
| the YANG module file into the I-D source markdown by reference instead of by | ||
| copy/paste, it becomes easier and more robust to ensure the actual .yang | ||
| files and the text of the draft don't accidentally diverge. | ||
|
|
||
| Likewise, examples are incredibly useful for understanding how the yang | ||
| model is best used, and incredibly easy to diverge in the course of | ||
| refactoring the YANG model. | ||
|
|
||
| It's hoped that by integrating support for external YANG files, the | ||
| quality of drafts that include YANG modules will improve with reduced | ||
| effort for authors. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,6 +6,10 @@ | |
| .tags | ||
| *~ | ||
| *.swp | ||
| .*.sw? | ||
| *.withyang | ||
| *.yangdeps | ||
| modules/ | ||
| /*-[0-9][0-9].xml | ||
| .refcache | ||
| .targets.mk | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,177 @@ | ||||||
| #!/usr/bin/env bash | ||||||
|
|
||||||
| input_md=$1 | ||||||
|
|
||||||
| if [ ! -f "$input_md" ] ; then | ||||||
| echo "usage: $0 <input.md> :" | ||||||
| echo " inserts yang into input.md+validates if VALIDATE_YANG=1" | ||||||
| echo " YANG-MODULE <mod.yang> (pyang lint)" | ||||||
| echo " YANG-TREE <mod.yang> (pyang lint)" | ||||||
| echo " YANG-DATA <mod.yang> <example.json> (yanglint validate json)" | ||||||
| echo " uses rsync to download imported modules to modules/ from iana" | ||||||
| echo "error: no input file: '$input_md'" | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| modules=$(grep YANG-MODULE $input_md | awk '{print $2;}' | tr "\n" " ") | ||||||
| tree_modules=$(grep YANG-TREE $input_md | awk '{print $2;}' | tr "\n" " ") | ||||||
| data_modules_str=$(grep YANG-DATA $input_md | awk '{print $2;}' | tr "\n" " ") | ||||||
|
|
||||||
| data_json_str="$(grep YANG-DATA $input_md | awk '{print $3;}' | tr "\n" " ")" | ||||||
| all_modules="$(echo "$modules $tree_modules $data_modules_str" | tr " " "\n" | sort | uniq | tr "\n" " " | sed -e 's/^ *//' | sed -e 's/ *$//')" | ||||||
|
|
||||||
| data_jsons=(); for d in $data_json_str; do data_jsons+=("$d"); done | ||||||
| data_modules=(); for d in $data_modules_str; do data_modules+=("$d"); done | ||||||
|
|
||||||
| #echo "input=$input_md, data='${#data_jsons[@]}: $data_jsons', mods='$all_modules'" | ||||||
|
|
||||||
| had_error=0 | ||||||
|
|
||||||
| target_xml="$(echo ${input_md} | sed -e 's/.md$/.xml/')" | ||||||
| for ((i=0; i < ${#data_jsons[@]}; i++)); do | ||||||
| if [ ! -f "${data_jsons[i]}" ]; then | ||||||
| echo "no example data file ${data_jsons[i]}" | ||||||
| had_error=1 | ||||||
| fi | ||||||
| done | ||||||
| for mod in ${all_modules}; do | ||||||
| if [ ! -f ${mod} ]; then | ||||||
| echo "no module file ${mod}" | ||||||
| had_error=1 | ||||||
| fi | ||||||
| done | ||||||
|
|
||||||
| if [ "${had_error}" != "0" ]; then | ||||||
| echo "error: some files not found, exiting" | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| fatal=0 | ||||||
| if ! which pyang 2>&1 > /dev/null ; then | ||||||
| echo "error in $0: pyang required; please install\n(pip install pyang)" | ||||||
| fatal=1 | ||||||
| fi | ||||||
|
|
||||||
| if ! which yanglint 2>&1 > /dev/null ; then | ||||||
| echo "error in $0: yanglint required; please install libyang:" | ||||||
|
There was a problem hiding this comment. This could download and install yanglint itself if it was missing. That would help in CI. There was a problem hiding this comment. That seems a bit rude and maybe requires privileges to add something in PATH. I guess maybe install into an environment variable location when it's missing and look for it there? There was a problem hiding this comment. I wasn't suggesting that you put this in /usr/local or anything like that. But you can download and compile the tool and run that copy. You can see an example of that sort of process in https://github.com/w3c/push-api/blob/gh-pages/Makefile |
||||||
| echo "(install libyang-dev or equivalent if available, or build:)" | ||||||
| echo " git clone https://github.com/CESNET/libyang" | ||||||
| echo " mkdir libyang/build" | ||||||
| echo " pushd libyang/build" | ||||||
| echo " cmake .." | ||||||
| echo " make && sudo make install" | ||||||
| echo " popd" | ||||||
| echo " yanglint --help" | ||||||
| echo "if it can't load libyang.so, try something like:" | ||||||
| echo " cmake -DCMAKE_INSTALL_PREFIX=$$HOME/local-installs -DCMAKE_INSTALL_RPATH=$$HOME/local-installs/lib .." | ||||||
| fatal=1 | ||||||
| fi | ||||||
|
|
||||||
| if [ "$fatal" != "0" ]; then | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| set -e | ||||||
|
|
||||||
| if [ ! -d modules/ ]; then | ||||||
| mkdir -p modules/ | ||||||
| fi | ||||||
|
|
||||||
| before=$(ls modules/ | wc -l) | ||||||
| after=${before} | ||||||
| last=$((before-1)) | ||||||
|
|
||||||
| echo "checking for module dependencies (downloading into modules/...)" | ||||||
| while [ "${last}" != "${after}" ]; do | ||||||
| last=${after} | ||||||
| for mod in ${all_modules}; do | ||||||
| missing=$( ( pyang --verbose --path modules:. ${mod} 2>&1 || true ) | \ | ||||||
| grep "error: module" | \ | ||||||
| sed -e 's/.*: error: module "\([^"]*\)" not found in search path.*/\1/') | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| if [ "$missing" != "" ]; then | ||||||
| if ! which rsync 2>&1 > /dev/null ; then | ||||||
|
There was a problem hiding this comment. The documentation does not mention a dependency on rsync. |
||||||
| echo "error in $0: rsync required; please install" | ||||||
| fi | ||||||
|
|
||||||
| for mis in ${missing}; do | ||||||
| bash -e -x -c "rsync -cvz rsync.iana.org::assignments/yang-parameters/${mis}*.yang modules/" | ||||||
|
There was a problem hiding this comment. This script forks a lot of bash subscripts unnecessarily. That will result in bad quoting. For instance, if There was a problem hiding this comment. I can use rsync directly, sure. I was doing this mainly so that when rsync errors, the call that caused it will be visible, but the quoting is a very good point and I think it should echo instead, thanks. |
||||||
| done | ||||||
| fi | ||||||
|
|
||||||
| # sample: | ||||||
| # err : Importing "ietf-routing-types" module into "ietf-dorms" failed. | ||||||
|
|
||||||
| # unfortunately, you can't distinguish between missing and erroring, | ||||||
| # so in particular when multiple local modules import each other but | ||||||
| # the bottom one is missing things, skip pulling local stuff, and | ||||||
| # instead re-check those. TBD: set them up as dependencies? | ||||||
|
|
||||||
| # TBD: maybe better to use yangcatalog.org to support I-D refs? | ||||||
| # see, e.g.: | ||||||
| # https://www.yangcatalog.org/yang-search/[email protected] | ||||||
| missing=$( ( yanglint -f json -D -V -p . -p modules ${mod} 2>&1 || true ) | \ | ||||||
| grep "err : Importing " | \ | ||||||
| sed -e 's/err : Importing "\([^"]*\)" module .* failed./\1/') | ||||||
| locals="" | ||||||
| while [ -f ${missing}.yang ]; do | ||||||
| already=0 | ||||||
| for loc in ${locals}; do | ||||||
| if [ "${loc}" = "${missing}.yang" ]; then | ||||||
| already=1 | ||||||
| fi | ||||||
| done | ||||||
| if [ "${already}" != "0" ]; then | ||||||
| break | ||||||
| fi | ||||||
| locals="${locals} ${missing}.yang" | ||||||
| missing=$( ( yanglint -f json -D -V -p . -p modules ${mod} 2>&1 || true ) | \ | ||||||
|
There was a problem hiding this comment. This looks like code repetition. Use a function perhaps. |
||||||
| grep "err : Importing " | \ | ||||||
| sed -e 's/err : Importing "\([^"]*\)" module .* failed./\1/') | ||||||
| done | ||||||
| if [ "$missing" != "" ]; then | ||||||
| for mis in ${missing}; do | ||||||
| if [ -f "$mis.yang" ]; then | ||||||
| missing=$( ( yanglint -f json -D -V -p . -p modules ${mis}.yang 2>&1 || true ) | \ | ||||||
| grep "err : Importing " | \ | ||||||
| sed -e 's/err : Importing "\([^"]*\)" module .* failed./\1/') | ||||||
| break | ||||||
| fi | ||||||
| done | ||||||
|
|
||||||
| if ! which rsync 2>&1 > /dev/null ; then | ||||||
| echo "error in $0: rsync required; please install" | ||||||
| fi | ||||||
| for mis in ${missing}; do | ||||||
| bash -e -x -c "rsync -cvz rsync.iana.org::assignments/yang-parameters/${mis}*.yang modules/" | ||||||
| done | ||||||
| fi | ||||||
| done | ||||||
| after=$(ls modules/ | wc -l) | ||||||
| done | ||||||
|
|
||||||
| echo "checking modules..." | ||||||
| for mod in ${all_modules}; do | ||||||
| if ! bash -x -e -c "pyang -E --verbose --ietf --lint --max-line-length 72 --path modules:. ${mod}" ; then | ||||||
| had_error=1 | ||||||
| fi | ||||||
| done | ||||||
|
|
||||||
| echo "checking examples..." | ||||||
| for ((i=0; i < ${#data_modules[@]} && i < ${#data_jsons[@]}; i++)); do | ||||||
| # TBD: ideally, yanglint would have a "treat warnings as errors" mode, | ||||||
| # which ideally would be used here. | ||||||
| # TBD: ideally, also [email protected] would not contain | ||||||
| # a redundant version that always reports this warning: | ||||||
| # warn: Module's revisions are not unique (2018-06-28). | ||||||
| bash -x -e -c "yanglint -f json -t data -D -s -V -p modules -p . -o /dev/null ${data_modules[i]} ${data_jsons[i]}" 2>&1 | grep -v "warn: Module's revisions are not unique (2018-06-28)" | ||||||
| if [ "${PIPESTATUS[0]}" != "0" ]; then | ||||||
| had_error=1 | ||||||
| fi | ||||||
| done | ||||||
|
|
||||||
| if [ "${had_error}" != "0" ]; then | ||||||
| echo "encountered yang errors in ${input_md}" | ||||||
| exit 1 | ||||||
| fi | ||||||
| echo "yang passed checks in ${input_md}" | ||||||
|
|
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wrap text to 80 columns.