Creates stats file of all repos

[sverhoeven]

Write a Typescript function that can parse out/*/*/evaluation-result.yml and stores in JSON in index-<timestamp>.json with the following info:

Scanned revision 008552b7266e229bd38553631d7dfe3554df18b2 of Git repository https://github.com/fair-software/howfairis.git
Found 1 files defining 17 unique dependencies within 1 scopes and 2 dependency levels
Detected 12 declared licenses
Completed scan with 9 unresolved policy violations and 0 issues

Would be nice to also include a link to out/*/*/scan-report-web-app.html report for each repo.

The file should also be stored as index-latest.json so Vue app always shows latest stats (#36)

[sverhoeven]

The yaml file can be parsed with https://www.npmjs.com/package/js-yaml which should already be listed as a dep in package.json.

As we are using Typescript we probably also need to add @types/js-yaml to devDependencies

[sverhoeven]

We should have unit test for different results like

• failed analyze
• no deps found
• multi scope
• etc.

[jspaaks]

Maybe something like this?

[{
    "fair-software": [{
        "fairtally": {
            "files": 1,
            "dependencies": 17,
            "detected": 12,
            "unresolved": 9,
            "issues": 0
        }
    }, {
        "howfairis": {
            "files": 1,
            "dependencies": 17,
            "detected": 12,
            "unresolved": 9,
            "issues": 0
        }

    }],
    "tortellini-tools": [{
        "action": {
            "files": 1,
            "dependencies": 17,
            "detected": 12,
            "unresolved": 9,
            "issues": 0
        }
    }]
}]

[jspaaks]

Changed my mind, I now prefer to have a flat structure

schema:

{
    "items": {
        "additionalProperties": false,
        "properties": {
            "dependencies": {
                "type": "number"
            },
            "detected": {
                "type": "number"
            },
            "files": {
                "type": "number"
            },
            "issues": {
                "type": "number"
            },
            "owner": {
                "type": "string"
            },
            "repo": {
                "type": "string"
            },
            "report": {
                "type": "string"
            },
            "unresolved": {
                "type": "number"
            },
            "url": {
                "type": "string"
            }
        },
        "type": "object",
        "required": [
            "dependencies",
            "detected",
            "files",
            "issues",
            "owner",
            "repo",
            "unresolved",
            "url",
            "report"
        ]
    },
    "minItems": 1
}

example data:

[{
        "dependencies": 1,
        "detected": 1,
        "files": 4,
        "issues": 4,
        "owner": "afasfa",
        "repo": "afasfasdfadsa",
        "url": "https://234234.com/d",
        "report": "https://tortellini-bkt2-5uk3-fbnr.s3.eu-central-1.amazonaws.com/.tortellini/out/3D-e-Chem/knime-kripodb/scan-report-web-app.html",
        "unresolved": 5
    },
    {
        "dependencies": 1,
        "detected": 1,
        "files": 4,
        "issues": 4,
        "owner": "afasfa",
        "repo": "afasfasdfadsa",
        "url": "https://234234.com/d",
        "report": "https://tortellini-bkt2-5uk3-fbnr.s3.eu-central-1.amazonaws.com/.tortellini/out/3D-e-Chem/knime-kripodb/scan-report-web-app.html",
        "unresolved": 3
    }
]

[fdiblen]

Example node script to use encoded data to get statistics.
test.js

[sverhoeven]

Instead yaml use the html file and parse the blob using some html parsing package and use #35 (comment) snippet to parse blob

[sverhoeven]

See https://github.com/oss-review-toolkit/ort/blob/edbb46cb1dab1529d5ffb81cba18d365b98ef23e/reporter-web-app/src/sagas/index.js#L29-L51 how to get the blob from the HTML dom tree and parse it.

[sverhoeven]

The data we decoded from the HTML file looks like evaluated-model.json which can be generated with ... report -f EvaluatedModel ...

We could use https://github.com/oss-review-toolkit/ort/blob/master/reporter/src/funTest/assets/evaluated-model-reporter-test-expected-output.json as input for a unit test.

[sverhoeven]

Changed my mind, I now prefer to have a flat structure

schema:

{
    "items": {
        "additionalProperties": false,
        "properties": {
            "dependencies": {
                "type": "number"
            },
            "detected": {
                "type": "number"
            },
            "files": {
                "type": "number"
            },
            "issues": {
                "type": "number"
            },
            "owner": {
                "type": "string"
            },
            "repo": {
                "type": "string"
            },
            "report": {
                "type": "string"
            },
            "unresolved": {
                "type": "number"
            },
            "url": {
                "type": "string"
            }
        },
        "type": "object",
        "required": [
            "dependencies",
            "detected",
            "files",
            "issues",
            "owner",
            "repo",
            "unresolved",
            "url",
            "report"
        ]
    },
    "minItems": 1
}

example data:

[{
        "dependencies": 1,
        "detected": 1,
        "files": 4,
        "issues": 4,
        "owner": "afasfa",
        "repo": "afasfasdfadsa",
        "url": "https://234234.com/d",
        "report": "https://tortellini-bkt2-5uk3-fbnr.s3.eu-central-1.amazonaws.com/.tortellini/out/3D-e-Chem/knime-kripodb/scan-report-web-app.html",
        "unresolved": 5
    },
    {
        "dependencies": 1,
        "detected": 1,
        "files": 4,
        "issues": 4,
        "owner": "afasfa",
        "repo": "afasfasdfadsa",
        "url": "https://234234.com/d",
        "report": "https://tortellini-bkt2-5uk3-fbnr.s3.eu-central-1.amazonaws.com/.tortellini/out/3D-e-Chem/knime-kripodb/scan-report-web-app.html",
        "unresolved": 3
    }
]

Is the url is the repository url which was given in of the lines of the repositories input file?

The action does not know where its output will be hosted so instead of using a full URL in report value use a relative path like 3D-e-Chem/knime-kripodb/scan-report-web-app.html.

[jspaaks]

url: yes it's the full repo url that the user supplied. it gets here via gitrepo:GitRepo
report: yes should be relative url. Note that it includes out/ as well at the moment (but not .tortellini anymore)

[sverhoeven]

The PR #126 has been merged giving a overview page showing statistics (nr of violations, deps, licenses, etc.) for each repo.

However the json files as described in this issue have not been created, because for mvp the index.html with JSON data inside was enough.
Split of #134 and #135 to handle JSON files.