Testing framework

[qku]

Hello, I am curious if somebody already thought about how to implement testing for qudi-core. There a few tests of fitting models in the tests folder right now based onunittest (they fail btw). Are there reasons for choosing unittest over pytest ? I am still rather new to testing but it looks to me like pytest is the de-facto standard here.

[qku]

The tool nox can help us to automate testing for different python versions. And the plugin pytest-cov let’s you know what portion of the code is already covered by the tests.

Is there a way to specify dependencies required only for development/testing in setup.py? I know a feature like this from poetry which makes packaging and releasing rather straightforward.

I followed the blog post Hypermodern Python and really liked using these tools.

[Neverhorst]

Hi @qku ,

as you have already noticed, the tests folder of qudi-core is currently not in use. Since I was writing and maintaining the qudi-core repo basically alone for most of the time, I simply never found the time to properly implement unit tests. The close collaboration with the Ulm University Institute for Quantum Optics and NVision Imaging Technologies GmbH made it quite easy to identify and eliminate bugs "in production".
I just put the tests directory in the repository tree along with a first draft on how a unit test could look based on my research into the topic from several years ago as a reminder/placeholder.

Proper deployment (not distribution) and missing unit tests have been a thorn in my side ever since the release of qudi-core because of the considerable effort it would take to set this up in a robust, maintainable and reasonably future-proof fashion. I would be more than happy to get some help on that front. 😃

For the choice of tools in general I would be very careful about regarding information from a single source as universal truth on the matter, regardless on how convincing it might seem. I have been maintaining Python projects (especially qudi) for over 10 years now and there have been some harsh learnings along the way...
With a popular tool like Python there is always an over-abundance of independent tools. Most of them don't keep up the pace of the fast moving community unfortunately. They often either get abandoned by their maintainers and/or stick to design choices that make them diverge from the Python canon (i.e. PEPs) causing them to become increasingly incompatible and harder to feature/support.
This is why I learned to appreciate packages maintained by the official Python work groups and standard library tools. While these packages often offer less functionality and ease-to-use (are less "sexy"), they tend to just work for a much longer period of time without requiring constant maintenance and are mostly well documented.
Python moves so fast, that information on good practices and tools you learned often get outdated within very few years. Believe me... I've re-learned a lot of things over the years because it became outdated.
That's why I chuckled a bit over the term "Hypermodern Python", because this title will age like fine milk in the sun. 😜

I'm not against third-party tooling in general, I just take the "official" tools as a safe bet until I have done extensive research and found something better (to maintain). In the end it always comes down to a balance between maintainability (very limited workforce) and functionality (alway check if you actually need the feature or if it is just appealing to your tech enthusiasm).

Regarding the capabilities of setup.py:
I agree that this way of configuring packaging is a bit outdated nowadays (but still supported) and should be replaced at some point with pyproject.toml (PEP 517 and PEP 660) to allow for more modern packaging and deployment tools. However I do not see a good reason for now to abandon setuptools, which is maintained by the PyPA and fully supports pyproject.toml configuration.
Testing build setups are anyway handled differently depending on what tooling is used.

TL;DR
If you want to prototype a testing workflow for qudi-core, by all means, please do so! 😃
How this is then packaged can be cleared up as a second step.
I'm happy to be of assistance when it comes to discussing design choices and CI/CD etc..

[prithviulm]

Hi guys, I am attemping to work on this. I am working on automatically building documentation from docstrings right now and plan to add unit tests and the whole CI/ CD stuff eventually. I am working on these on a fork, https://github.com/prithviulm/qudi-core. Any help/ suggestions/ comments would be appreciated!

[krokosik]

@prithviulm That's great news. I would be happy to help and hear how is the work coming along?

[prithviulm]

Thanks for your interest @krokosik! We are nearly ready to merge the first set of tests and documentation into the main branch. Most of the docs stuff lives here and most of the tests stuff lives here as of now.

We hope these initial tests and documentation serve as a useful framework to build upon after they are merged. There is still a lot of work to be done after this initial step so it would be a great time to join the effort.

With regards to tests, qudi-core is difficult to write unit test for because the different submodules cannot be easily instantiated without having to run the entire qudi-core process. An overhaul of this to enable unit-testing would be beneficial especially as we would like to move to pyside6 at some point and having unit tests would help, but this could be quite challenging. We also intend to have extensive tests for the qudi-iqo-modules to test the various logic files and other things. The tests we will merge soon include a few such tests but we definitely need more.

With regards to documentation, we have so far tried to standardise the existing docstrings and documentation (to be merged soon) such that it can render on a readthedocs page nicely. We need to add some additional documentation to better guide (new) users on how to use/ configure/ modify qudi. There are also places where documentation and docstrings could be entirely missing.

Let us know what interests you and if you have any other questions. We can also discuss this further during the software meetings or separately.

[krokosik]

@prithviulm ah sorry for the late reply! I am just starting implementing qudi in our lab, so will start with your fork and let you know the results :)

[prithviulm]

That's great to hear! If you're installing Qudi to use for lab use, I'd definitely recommend installing it from the ulm-iqo repository and not from my fork. My fork has not been synced in a while as we're still resolving some issues with tests and documentation, so some of the latest bug fixes are not available on my fork yet.