The tests are cruddy

[DylanLacey]

Hey @bootstraponline and other interested parties, I want to pick your brains.

There's a lot of work to be done on the gems, and I'm super keen to have at it, but I know at the moment the tests aren't in a state which allows for that to happen easily.

What I think makes the most sense, is:

1. Write 'dummy' Selenium servers and REST endpoints
2. Refactor existing tests into their own repo/s and include them as submodules
3. Configure existing tests for integrations to use the dummy servers, and just run some sample tests; Assert against the dummy SE servers and REST endpoints to make sure they ran with expected caps
4. Be more smarterer about the state of the SC binary using process checks and the REST API to determine if tunnels are up, and monitoring console output to check quiet et al, for the SC test suite
5. Remove every test that's about the gem's internal state because who cares. It's what it does to a test framework that's important.

I think that makes it most possible to do integration tests, test the documentation and all relevant functionality, without having to spin up millions of Sauce Labs sessions or mock AllTheThings.

Opinions gratefully sought.

[bootstraponline]

In general I like the Google recommendation of:

70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests
http://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html

I can see having a single Sinatra server which mocks out the REST endpoints as an alternative to webmock. The downside is it'll be slower than webmock or other lower level mocking solutions. I guess the advantage would be it's easier to maintain and write tests against. I don't think we'd want multiple mock servers, ideally one would be sufficient.

In one of the projects I've been working on, we have a server description in markdown and then the mock server is automatically generated from the markdown file via api-mock. That means adding new endpoints or updating the data is as simple as modifying the markdown. Might be a bit much for this specific use case though.

I think each gem should be in its own repo. The mock server can be published as a gem and then added as a gem dependency (git submodules are not fun). Also each gem should have proper unit tests in addition to the integration tests that run automatically on PRs via Travis. It'd be nice to have rubocop setup as well for consistency in code style (we just finished this effort on the appium Ruby gem).

SC gem is super problematic so it'd definitely benefit from a TDD style approach. This is where a mock server isn't that useful though because fundamentally it's about spinning up exactly one SC process and then making sure it's terminated after tests complete. Process control is best done via unit tests. E2E tests aren't going to detect infinite loops or crashing on certain command line arguments.

I think we actually need more unit tests about the internal state, not less. That's in addition to better E2E tests. I agree that manually mocking everything or using lots of Sauce Labs sessions isn't the way to go. I hope this feedback is useful. I think it's great that Sauce is going to invest engineering resources in making the Ruby integration better.