Add parent dependency check on samples

[michael-valdron]

What does this PR do?:

Summarize the changes. Are any stacks or samples added or updated?

Adds new script and new workflow to perform testing on child samples of stacks.

Which issue(s) this PR fixes:

Link to github issue(s)

fixes devfile/api#1142

PR acceptance criteria:

• Contributing guide
  Have you read the devfile registry contributing guide and followed its instructions?
• Test automation
  Does this repository's tests pass with your changes?
• Documentation
  Does any documentation need to be updated with your changes?
• Check Tools Provider
  Have you tested the changes with existing tools, i.e. Odo, Che, Console? (See devfile registry contributing guide on how to test changes)

How to test changes / Special notes to the reviewer:

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: michael-valdron

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [michael-valdron]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[michael-valdron]

Workflow tested on the main branch on my folk: https://github.com/michael-valdron/devfile-registry/actions/runs/5682241491/job/15400190524

[michael-valdron]

Workflow tested on the main branch on my folk: https://github.com/michael-valdron/devfile-registry/actions/runs/5682241491/job/15400190524

Tied branch now called main-3464e43, main has been restored as repository upstream.

[thepetk]

I've tried to test the script locally following the instructions:

• Upon root dir of project I had this error:

$ STACKS_DIR=samples/.cache bash tests/validate_devfile_schemas.sh

+ stacksDir=samples/.cache
+ stackDirs='nodejs-basic code-with-quarkus go-basic java-springboot-basic python-basic'
+ ginkgo run --procs 2 tests/validate_devfile_schemas -- -stacksPath samples/.cache -stackDirs 'nodejs-basic code-with-quarkus go-basic java-springboot-basic python-basic'
Failed to compile validate_devfile_schemas:

go: cannot find main module, but found .git/config in <mypath>/michael_valdron/devfile-registry
        to create a module there, run:
        go mod init

Ginkgo ran 1 suite in 12.799434ms
Test Suite Failed

[michael-valdron]

@thepetk What is your output for go version?

[michael-valdron]

Made changes 7c0d08c so STACKS_DIR can be either a relative or absolute path when executing validate_devfile_schemas.sh.

[thepetk]

@thepetk What is your output for go version?

@michael-valdron output is
go version go1.19.1 linux/amd64

[michael-valdron]

@thepetk What is your output for go version?

@michael-valdron output is go version go1.19.1 linux/amd64

@thepetk try running it again with a go 1.18.x runtime, go 1.19.1 could be having issues running it.

[thepetk]

Finally got it run successfully :)

One thing I had to do other than those steps, was to add a go.work file in my repo's root dir.

It seems that is working fine :)

One test I tried to make is to update the go-basic sample inside the extraDevfileEntries. I've tried to include multiple sample versions:

  - name: go-basic
    displayName: Basic Go
    description: Go 1.16 application
    icon: https://go.dev/blog/go-brand/Go-Logo/SVG/Go-Logo_Blue.svg
    tags:
      - Go
    projectType: Go
    language: Go
    provider: Red Hat
    versions:
      - version: 1.0.2
        schemaVersion: 2.1.0
        git:
          remotes:
            origin: https://github.com/devfile-samples/devfile-sample-go-basic.git
      - version: 2.1.0
        schemaVersion: 2.2.0
        default: true
        git:
          checkoutFrom:
            revision: update_devfile
          remotes:
            origin: https://github.com/thepetk/devfile-sample-go-basic.git

In the parent file I had:

  - name: go
    children:
      - name: go-basic
        version: 1.0.2
  - name: go
    version: 2.1.0
    children:
      - name: go-basic
        version: 2.1.0

(Which is ok)

During testing I was expecting to see the test fail for one of the golang children (go-basic version 2.1.0). Although I had an issue with the quarkus. The trace of the test was:

• [FAILED] [1.984 seconds]
validate stacks follow the schema [It] stack: code-with-quarkus
/home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:50

  Timeline >>
  {/home/tpetkos/github/michael_valdron/devfile-registry/samples/.cache/go-basic/2.1.0/devfile.yaml  [] <nil> <nil> []  <nil> <nil> map[] <nil>}
  [FAILED] in [It] - /home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:61 @ 08/01/23 17:18:55.812
  << Timeline

  [FAILED] Expected
      <*errors.withStack | 0xc0004dde60>: {
          error: <*errors.withMessage | 0xc000693e40>{
              cause: <*multierror.Error | 0xc000693e20>{
                  Errors: [
                      <*errors.errorString | 0xc000450cd0>{
                          s: "Some Commands are already defined in parent: build-image, deploy, deployk8s. If you want to override them, you should do it in the parent scope.",
                      },
                  ],
                  ErrorFormat: nil,
              },
              msg: "failed to populateAndParseDevfile",
          },
          stack: [0x14db668, 0x14e2e38, 0x156ad58, 0x1530e1b, 0x1543958, 0x46d841],
      }
  to be nil
  In [It] at: /home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:61 @ 08/01/23 17:18:55.812

[michael-valdron]

During testing I was expecting to see the test fail for one of the golang children (go-basic version 2.1.0).
Although I had an issue with the quarkus. The trace of the test was:

• [FAILED] [1.984 seconds]
validate stacks follow the schema [It] stack: code-with-quarkus
/home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:50

  Timeline >>
  {/home/tpetkos/github/michael_valdron/devfile-registry/samples/.cache/go-basic/2.1.0/devfile.yaml  [] <nil> <nil> []  <nil> <nil> map[] <nil>}
  [FAILED] in [It] - /home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:61 @ 08/01/23 17:18:55.812
  << Timeline

  [FAILED] Expected
      <*errors.withStack | 0xc0004dde60>: {
          error: <*errors.withMessage | 0xc000693e40>{
              cause: <*multierror.Error | 0xc000693e20>{
                  Errors: [
                      <*errors.errorString | 0xc000450cd0>{
                          s: "Some Commands are already defined in parent: build-image, deploy, deployk8s. If you want to override them, you should do it in the parent scope.",
                      },
                  ],
                  ErrorFormat: nil,
              },
              msg: "failed to populateAndParseDevfile",
          },
          stack: [0x14db668, 0x14e2e38, 0x156ad58, 0x1530e1b, 0x1543958, 0x46d841],
      }
  to be nil
  In [It] at: /home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:61 @ 08/01/23 17:18:55.812

This is an interesting result, it is reporting the stack as being code-with-quarkus but the error in the test seems to be the expected one with go-basic 2.1.0, I'll need to check if there is a problem with the inputs or if there is a bug within this test suite which is causing the wrong stack name to be reported.

[openshift-merge-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@michael-valdron: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/v4.13-images	7c0d08c	link	true	/test v4.13-images
ci/prow/v4.13-registry-test	7c0d08c	link	true	/test v4.13-registry-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[michael-valdron]

Closing due to age, will re-open when focus to devfile/api#1142 is resumed.

[michael-valdron]

Resuming review of changes here: #397