Can I recycle the same app instance between multiple tests?

[adrian-moisa]

I'm planning to write several hundred tests. I noticed that on average every patrolTest adds 3-4 secs of run time in my setup. The app seems to be relaunched on every test. An additional delay seems to be happening due to the auth system waiting for keycloak to confirm the user is logged in. Both added they slow down the tests. Now for a few dozen tests it's not terrible but I'm planning to run several hundred, even potentially 1000 tests in the future. I'm interested to recycle the same app instance between tests without merging all expect assertions in on giant patrolTest.

I'm completely fine with the fact that test could end up cross-talking between them. I've crafted a carefully orchestrated sequence of events in the my app and I'm totally ok to recycle the same instance between all tests. As a matter of fact I'd rather have it done this way to ensure long running sessions wont have some fancy bug hidden inside. By recycling the same instace of the app I'd expect to see a massive speed boost in the tests run time. Having this speed boost is critical for my usecase. I want to have the team run these tests quite often, therefore the shorter the run time the more likely the team will do this effort consistently. Thus we get faster notice if something breaks.

#1413 - I've seen this conversation but it appears to reference an older issue where the app was rebuilt for each test.

[adrian-moisa]

Any advice on this one? Would be a major improvement. Thank you!

[bartekpacia]

Hi again @adrian-moisa, sorry for our lack of response.

For future readers: this question has been answered in our Discord server (link)

[adrian-moisa]

I've been able to achieve this result by merging all tests in one single master test. All the sub tests are function calls each one with a print statement. This also helps me visualise how far the tests have progressed. The tests are grouped by modules. Before tests of each module I trigger one navigation event. Every single test is expected to restore the app to previous condition when making changes. Therefore we are able to execute any of the tests in isolation or in random groupings without changing the final behaviour. It's also nice improvement since now we are able to change quickly which tests are run in hot reload mode. Again a great improvement in dev life quality.

I know this approach goes against the grain/convention, but it yields great results for us. From 24mins of run time we are now down to 4:30. Massive improvement. Also I'm really happy that I've been able to catch several bugs that happen only in complex interactions between pages. The app now is stable and steady enough to be able to go trough over 300 tests in one single instance with no crash. Again, really happy to verify and prove that the app can run for long sessions without crashing. Thanks to e2e testing with Patrol we get a lot of bang for buck spent testing. Now I'm working on a way to trigger page reload and cache regeneration to cover the be code as well with the same tests.

Take away, the ability to configure if tests reuse the same instance would be a nice improvement, but it can also be done by merging everything in one large test. The only drawback of this approach is that the test run will stop at the first broken one. So you can't get a clear indication how many are down in case a refactoring broke more tests. The best you can to get an estimate/survey is to disable them one by one. However this is not a big pain for now since we do a lot of effort in planning ahead and using clean code practices. Therefore we don't have such catastrophic outcomes often.