Skip to login in finish step if '-t' is specfied

[idorax]

If option --test (-t) is specified, we should skip to log into the guest in finish phase as it has been done already.

Fix: #1918

[lukaszachy]

Unfortunately this breaks --step and --when functionality:
Same reproducer.fmf with tmt run login -t --step prepare - As a user I might want to login to the system before the execute step.

The --when breakage is questionable, however the main difference between --test and --when is login to 'execute' versus 'finish' step, which have different environment.
With that said I think there might be some valid use case to user -t and --when together.

[idorax]

Unfortunately this breaks --step and --when functionality ...

@lukaszachy, how about introducing a variable to record the action to log into the guest is done? If it's done, skip to the login action in finish step.

[lukaszachy]

Unfortunately this breaks --step and --when functionality ...

@lukaszachy, how about introducing a variable to record the action to log into the guest is done? If it's done, skip to the login action in finish step.

IMO it is not necessary. Your approach is good, just stop the login in finish only if there isn't when and step used. Current patch is too eager to stop the login.

From the options --command doesn't affect whether the login should happen or not, but --test, --step and --when do.

[idorax]

Unfortunately this breaks --step and --when functionality ...

@lukaszachy, how about introducing a variable to record the action to log into the guest is done? If it's done, skip to the login action in finish step.

IMO it is not necessary. Your approach is good, just stop the login in finish only if there isn't when and step used. Current patch is too eager to stop the login.

@lukaszachy, to check when or step used, looks we can keep using opt('when') or opt('step').

$ find tmt -name "*.py" | xargs egrep -- "'--when|'--step"
tmt/steps/__init__.py:            '-s', '--step', metavar='STEP[:PHASE]', multiple=True,
tmt/steps/__init__.py:            '-s', '--step', metavar='STEP[:PHASE]', multiple=True,
tmt/steps/__init__.py:            '-w', '--when', metavar='RESULT', multiple=True,

Would you please help to paste a user case having both -t and --when or -t, --when and --step?

[lukaszachy]

Would you please help to paste a user case having both -t and --when or -t, --when and --step?

How about the opposite question - Why should we disallow that?

I've realized that '-c CMD' can be used to workaround fact one cannot append 'prepare' or 'finish' step from the commandline (it would replace step definition from the plan) to do something on the guest.
But I'm not sure how well one can combine options (e.g. tmt run ... login -c 'date' login -t login --step prepare -c 'some prep magic' ... ). I'll take a look later if you don't find it sooner.

[lukaszachy]

While trying to create use case for login -t --when I've realized that it behaves a bit surprising.

Current implementation is that -t logs into execute right after the test finishes and has same environment as the test (very easy to rerun the test) while --when logs into finish depending on the execute outcome.

IMO we could keep login --when RESULT work as is (login into finish step depending on the execute) however the login -t --when RESULT should be changed to "log into execute right after the test depending on the test RESULT".

This would cover use case when User has several tests in the plan but want to login immediately after RESULT (usually fail or error) to investigate what has happened. Currently it is not easy as -t logs in after each test and login in finish is too late as later tests might change the SUT significantly.

WDYT @psss @happz @adiosnb @idorax ?

[thrix]

/packit test

[adiosnb]

While trying to create use case for login -t --when I've realized that it behaves a bit surprising.

Current implementation is that -t logs into execute right after the test finishes and has same environment as the test (very easy to rerun the test) while --when logs into finish depending on the execute outcome.

IMO we could keep login --when RESULT work as is (login into finish step depending on the execute) however the login -t --when RESULT should be changed to "log into execute right after the test depending on the test RESULT".

This would cover use case when User has several tests in the plan but want to login immediately after RESULT (usually fail or error) to investigate what has happened. Currently it is not easy as -t logs in after each test and login in finish is too late as later tests might change the SUT significantly.

WDYT @psss @happz @adiosnb @idorax ?

Yeah, for me it makes sense to be able to combine these parameters to be able to login to a server after each failed or error test.

+1 for me

[lukaszachy]

This is the usual 'issue looks easy at first but later it explodes' example.

Conclusion from the hacking session:
login -t --when RESULT should login to the guest immediately after the test (so it has to be in the execute) if the test ended with RESULT . No additional login in the finish should happen unless requested.

We allow multiple --step and --when so one way to implement this could be to implicitly use --step execute if -t is used.
So login -t --when RESULT would be the same as login -t --when RESULT --step execute thus the login in finish will not happen.
However user can request login -t --when RESULT --step finish and because of the -t it would be as if login -t --when RESULT --step finish --step execute was used and login will happen in the execute (after each test with RESULT) and finish (if any test has RESULT). IF user doesn't want to login in execute for this command line they shouldn't specify -t option.

I'm not sure that mutiple login on the commandline work correctly so let's support single login command with several options.

This should ready for the future once multiple subcommands can be used properly (the logic of login command can stay the same).

[lukaszachy]

Would you please help to paste a user case having both -t and --when or -t, --when and --step

To finally reply the use cases:

-t --when : As tmt user I want to login to the guest with the same environment as the test immediately after that test finishes with RESULT so I can investigate why if finished in such result (e.g. --when fail to see why test fails)

-t --when --step: As a tmt user I want to investiage test result in execute step but I want to login to the guest in other steps than execute as well.

[lukaszachy]

My testing plan:

/p:
  execute:
    how: tmt
  discover:
    how: fmf
  provision:
    how: container

/t:
  /good:
    test: 'true'

  /bad:
    test: 'false'

The tmt run login -t works as expected -- logs in 'execute' step immediately after each test, no login in finish

But tmt run login -t --when fail however logs in 'finish' and there is no way to stop that (adding --step execute makes login happen at the end of 'execute' step (once all tests are finished')

Could the login in 'finish' be disabled when '-t --when' is set but '--step finish' is not?
So tmt run login -t --when fail will login only after '/t/bad' but I can ask tmt to login in finish as well by tmt run login -t --when fail --step finish

[adiosnb]

My testing plan:

/p:
  execute:
    how: tmt
  discover:
    how: fmf
  provision:
    how: container

/t:
  /good:
    test: 'true'

  /bad:
    test: 'false'

The tmt run login -t works as expected -- logs in 'execute' step immediately after each test, no login in finish

But tmt run login -t --when fail however logs in 'finish' and there is no way to stop that (adding --step execute makes login happen at the end of 'execute' step (once all tests are finished')

Could the login in 'finish' be disabled when '-t --when' is set but '--step finish' is not? So tmt run login -t --when fail will login only after '/t/bad' but I can ask tmt to login in finish as well by tmt run login -t --when fail --step finish

What do you think about extending the login test set with these cases?