katello tests fail on el8 box

[evgeni]

I've been poking Katello on EL8 (austin_powers-i_like_to_live_dangerously.jpg) and encountered the following bats errors:

    1..1
    ok 1 verify no old SCL packages
    1..3
    ok 1 # skip (FOREMAN_EXPECTED_VERSION is not set, not asserting) assert correct Foreman version is installed
    ok 2 check web app is up
    ok 3 check smart proxy is registered
    1..10
    ok 1 check smart proxy is registered
    ok 2 assert puppet version
    ok 3 wake up puppet agent
    ok 4 check host is registered
    ok 5 install puppet module
    ok 6 import motd puppet class
    ok 7 Assign environment to default taxonomies
    ok 8 # skip (Assignment not needed) Assign proxy to default taxonomies
    ok 9 assign puppet class to host
    ok 10 apply class with puppet agent
    1..2
    ok 1 check hammer ping
    ok 2 check service status
    1..57
    ok 1 Create an Empty Organization
    ok 2 create an Organization
    ok 3 create a product
    ok 4 create package repository
    ok 5 upload package
    ok 6 sync repository
    ok 7 create a file repository
    ok 8 sync file repository
    ok 9 fetch file from file repository
    ok 10 create a container repository
    ok 11 sync container repository
    ok 12 # skip (Puppet content is not available in scenarios without Pulp 2) create puppet repository
    ok 13 # skip (Puppet content is not available in scenarios without Pulp 2) upload puppet module
    ok 14 create lifecycle environment
    ok 15 create content view
    ok 16 add yum repo to content view
    ok 17 publish content view
    ok 18 promote content view
    ok 19 create activation key
    ok 20 disable auto-attach
    ok 21 add subscription to activation key
    ok 22 install subscription manager
    ok 23 disable puppet agent to prevent checkin from registering host to another org
    ok 24 delete host if present
    not ok 25 register subscription manager with username and password
    # (in test file fb-content-katello.bats, line 170)
    #   `subscription-manager register --force --org="${ORGANIZATION_LABEL}" --username=admin --password=changeme --env=Library' failed with status 70
    # rc=1
    # This system is currently not registered.
    # rc=0
    # All local data removed
    #
    # WARNING
    #
    # The yum/dnf plugins: /etc/dnf/plugins/subscription-manager.conf were automatically enabled for the benefit of Red Hat Subscription Management. If not desired, use "subscription-manager config --rhsm.auto_enable_yum_plugins=0" to block this behavior.
    # rc=0
    # Updating Subscription Management repositories.
    # Unable to read consumer identity
    # This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
    # No match for argument: katello-ca-consumer-*
    # No packages marked for removal.
    # Dependencies resolved.
    # Nothing to do.
    # Complete!
    # rc=1
    # curl: (22) The requested URL returned error: 404 Not Found
    # error: skipping http://localhost/pub/katello-ca-consumer-latest.noarch.rpm - transfer failed
    # Retrieving http://localhost/pub/katello-ca-consumer-latest.noarch.rpm
    # Registering to: subscription.rhsm.redhat.com:443/subscription
    # Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
    not ok 26 register subscription manager with activation key
    # (in test file fb-content-katello.bats, line 183)
    #   `subscription-manager list --consumed | grep "${PRODUCT}"' failed
    # rc=1
    # This system is currently not registered.
    # rc=0
    # All local data removed
    # rc=70
    # Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
    ok 27 start puppet again
    not ok 28 check content host is registered
    # (in test file fb-content-katello.bats, line 192)
    #   `hammer host info --name $HOSTNAME' failed with status 70
    # Error: host not found.
    not ok 29 enable content view repo
    # (in test file fb-content-katello.bats, line 196)
    #   `subscription-manager repos --enable="${ORGANIZATION_LABEL}_${PRODUCT_LABEL}_${YUM_REPOSITORY_LABEL}" | grep -q "is enabled for this system"' failed
    ok 30 install katello-host-tools
    not ok 31 install package locally
    # (from function `tIsRedHatCompatible' in file os_helper.bash, line 4,
    #  from function `tPackageInstall' in file os_helper.bash, line 89,
    #  in test file fb-content-katello.bats, line 205)
    #   `tPackageInstall walrus-0.71 && tPackageExists walrus-0.71' failed
    # Updating Subscription Management repositories.
    # Unable to read consumer identity
    # This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
    # Last metadata expiration check: 0:04:20 ago on Fri 11 Sep 2020 09:02:48 AM UTC.
    # No match for argument: walrus-0.71
    # Error: Unable to find a match: walrus-0.71
    not ok 32 check available errata
    # (in test file fb-content-katello.bats, line 213)
    #   `hammer host errata list --host $HOSTNAME | grep 'RHEA-2012:0055'' failed
    ok 33 # skip (katello-agent support is not available in scenarios without Pulp 2) install katello-agent
    ok 34 # skip (katello-agent support is not available in scenarios without Pulp 2) 30 sec of sleep for groggy gofers
    ok 35 # skip (katello-agent support is not available in scenarios without Pulp 2) install package remotely (katello-agent)
    ok 36 # skip (katello-agent support is not available in scenarios without Pulp 2) install errata remotely (katello-agent)
    ok 37 # skip (katello-agent support is not available in scenarios without Pulp 2) package remove (katello-agent)
    ok 38 # skip (Puppet content is not available in scenarios without Pulp 2) add puppet module to content view
    ok 39 # skip (Puppet content is not available in scenarios without Pulp 2) publish content view with puppet content
    ok 40 # skip (Puppet content is not available in scenarios without Pulp 2) promote content view with puppet content
    ok 41 try fetching container content
    not ok 42 create and sync modules-rpms repo
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 292)
    #   `tSkipIfHammerBelow018' failed
    not ok 43 create and sync rpm-deps repo
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 303)
    #   `tSkipIfHammerBelow018' failed
    not ok 44 create first component content view
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 313)
    #   `tSkipIfHammerBelow018' failed
    not ok 45 add yum and docker repos to first component content view
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 320)
    #   `tSkipIfHammerBelow018' failed
    not ok 46 add errata exclude filter to first component content view
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 341)
    #   `tSkipIfHammerBelow018' failed
    not ok 47 add package exclude filter to first component content view
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 350)
    #   `tSkipIfHammerBelow018' failed
    not ok 48 add module include filter to first component content view
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 359)
    #   `tSkipIfHammerBelow018' failed
    not ok 49 publish first component content view
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 370)
    #   `tSkipIfHammerBelow018' failed
    not ok 50 create composite content view
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 377)
    #   `tSkipIfHammerBelow018' failed
    not ok 51 publish and promote composite content view
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 388)
    #   `tSkipIfHammerBelow018' failed
    not ok 52 incremental update first component cv with composite propagation
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 397)
    #   `tSkipIfHammerBelow018' failed
    not ok 53 ensure component cv 1 version 1.1 has proper environments
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 407)
    #   `tSkipIfHammerBelow018' failed
    not ok 54 ensure composite cv version 1.1 has proper environments
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 417)
    #   `tSkipIfHammerBelow018' failed
    not ok 55 ensure component cv 1 latest version has proper content
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 428)
    #   `tSkipIfHammerBelow018' failed
    not ok 56 ensure component cv 2 latest version has proper content
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 451)
    #   `tSkipIfHammerBelow018' failed
    not ok 57 ensure composite cv latest version has proper content
    # (from function `tSkipIfHammerBelow018' in file foreman_helper.bash, line 32,
    #  in test file fb-content-katello.bats, line 474)
    #   `tSkipIfHammerBelow018' failed
    1..1
    ok 1 Delete an Organization
    1..1
    ok 1 collect important logs

I am filing this here and not on Katello as I think this is partially bad tests and not always a bug.

@jlsherrill @ehelms 👀

[evgeni]

the hammer018 errors should be gone after #1207
the registration 404 via theforeman/puppet-foreman_proxy_content#282