Wildfly version for tests

[radcortez]

For running tests in Jenkins we're using our own Wildfly build, based on the latest sources. Shouldn't we stick with the stable release versions and then update when a new version comes out?

A few tests at the moment are failing because of the Wildfly version. They run fine in 8.0.0.Final, but fail in 8.0.1.Final-SNAPSHOT. Since the failing version is a SNAPSHOT, the possible test issues may be solved before the final release and in that case we're just having erroneous information.

What do you thing @arun-gupta ?

[bartoszmajsak]

I think we should stick to what's available through release channels. We could even give 8.1.0.CR1 a try.

[noahwhite]

Good coverage: FCS
Better coverage: FCS and latest beta release (eg. CRx)
Best coverage: FCS, latest beta, and nightly.

[bartoszmajsak]

Forgive my ignorance, but what stands for FCS @noahwhite ?

[radcortez]

First Customer Shipment I think.

So, @noahwhite you're suggesting to run tests on all those versions?

[bartoszmajsak]

I learn sth everyday ;) Thx!

[noahwhite]

@bartoszmajsak Yes, FCS == First Customer Shipment of a particular release - so for example 8.0.0 Final looks like its the latest WF FCS build.

I don't know the WF release structure but it looks like you have Alpha, Beta, CR then FCS.

@radcortez Yes running the samples/tests in parallel with the 3 WF releases would give you the best coverage. You would know if recent commits to the trunk caused a regression in WF, or exposed a deficiency in your samples.

Running the samples against the latest non-nightly dev. release will do the same for that. Where the trunk results would be mostly useful to devs. these results would be useful to a wider audience of people who are using or testing these pre-release blds. If there's only a trunk run then folks using the pre-release blds (CR, Beta, Alpha) may run into unexpected issues that have been fixed in trunk. Having both allows you to see what those issues would be and allows you to see where the regressions and fixes were introduced between them.

Finally, running tests against FCS obviously lets users and devs. how the stable release plays with these samples.

On a side note, @arun-gupta and everyone who has contributed, these samples are a great resource for the Java EE community. Thanks for all you hard work on putting them together!

[bartoszmajsak]

Especially if we start considering these samples as some sort of "TCK" ;) But it's still a long journey to achieve it.

[aslakknutsen]

I agree with @noahwhite, Latest Final(s), Latest release and Nightly would be nice.

One part of having the samples is to show how to use JavaEE7, and one part of having them Arquillian tested is to show what works where. Basically the state of portability.

Hopefully we can catch some of the Nightly issues and report back upstream before they become Latest Release or Latest Final.

[arun-gupta]

When the project was created, WildFly was still under active development. So WildFly workspace was polled every 15 minutes, and if there were any changes then a new WildFly build was generated and that triggered a new test run. Now that WildFly 8 is final, here is what would make sense:

• Latest Final
• Latest non-Final release
• 15 minute polling to the workspace (or somehow if we can trigger the test build with every WildFly workspace commit)

As @aslakknutsen said, this will allow us to catch errors during development and hopefully fixed before the release become final.

Each test failure should either be filed as a test issue or an issue in WildFly.

Does that help ?

[radcortez]

Yes Arun. I'm thinking on how we should setup this in Jenkins:

• Keep the existing jobs.
• Add two additional jobs for javaee samples that run on Latest Final and Latest Non-Final.

Or

• Expand the javaee samples job and have everything there.

What do you guys think?

[arun-gupta]

I like the first proposal.

[radcortez]

So, I just added 2 new jobs Wildfly 8.0.0.Final and Wildfly 8.1.0.CR2. Still need a few improvements, but let me know what you think.

[arun-gupta]

Have you done an analysis of what brings the number of failures from 13 (https://arungupta.ci.cloudbees.com/job/javaee7-samples-on-wildfly-cb/) to 27 (https://arungupta.ci.cloudbees.com/job/Java%20EE%20Samples%20on%20Wildfly%208.0.0.Final/) ?

[radcortez]

No, I just added the build, but I'm having a look into it in the next few days.

[radcortez]

Ok, I did the analysis and the problem was a misconfiguration in the JMS for the new jobs. Now the failures are more alike.

Final and CR versions are having failures in a few JSF tests that don't happen in the SNAPSHOT version. I'll have a look when I have some time.

I guess we can close this issue?

[arun-gupta]

Open the bugs for discrepancies and close this one ?