Switch to Poetry

[Conan-Kudo]

This allows kiwi to be built, developed, tested, and installed through Poetry.

Fixes #2370.
Fixes #2369.

Changes proposed in this pull request:

• Add Poetry definition
• Remove setuptools definition
• Raise the minimum Python version to 3.9.

[Conan-Kudo]

This is nowhere near complete, because I'm still working on figuring out:

• Converting our optional dependencies into extras
• Handling our special sdist logic (if at all possible)
• Reworking our CI to use it
• Reworking our spec to use it
• Update contribution chapter in the docs
• Handle bumpversion properly?

[schaefi]

Thanks for starting this. I switched the branch and did some tests. I wondered about the entry-points. Usually I do this:

tox -e devel
. .tox/unit_py3_11/bin/activate
which kiwi-ng
/home/ms/Project/kiwi/.tox/unit_py3_11/bin/kiwi-ng
kiwi-ng --version
KIWI (next generation) version 10.0.0
deactivate

now the same with poetry

poetry shell
Spawning shell within /home/ms/.cache/pypoetry/virtualenvs/kiwi-Btua-i95-py3.11
. /home/ms/.cache/pypoetry/virtualenvs/kiwi-Btua-i95-py3.11/bin/activate
which python3
/home/ms/.cache/pypoetry/virtualenvs/kiwi-Btua-i95-py3.11/bin/python3
==> ok
which kiwi-ng
which: no kiwi-ng in ...
==> Hmm, no entry points created
exit

I wonder how you would call kiwi-ng inside of the venv without the entry-points. The entry-points are defined in setup.py as

'entry_points': {
        'kiwi.tasks': [
            'image_info=kiwi.tasks.image_info',
            'image_resize=kiwi.tasks.image_resize',
            'result_bundle=kiwi.tasks.result_bundle',
            'result_list=kiwi.tasks.result_list',
            'system_build=kiwi.tasks.system_build',
            'system_create=kiwi.tasks.system_create',
            'system_prepare=kiwi.tasks.system_prepare',
            'system_update=kiwi.tasks.system_update'
        ],
        'console_scripts': [
            'kiwi-ng=kiwi.kiwi:main',
            'kiwicompat=kiwi.kiwi_compat:main'
        ]
    },

It's really crucial that this stuff continues to work. Not only for the usability of a venv but also to not break kiwi's plugin support

[schaefi]

btw the kiwicompat entry point can be deleted, we won't need any old compat layer for 10.x.x

[Conan-Kudo]

With poetry, you initialize the virtualenv using poetry itself, and use poetry run to run the script.

[schaefi]

With poetry, you initialize the virtualenv using poetry itself, and use poetry run to run the script.

ah I see

poetry install
poetry run kiwi-ng --version
KIWI (next generation) version 10.0.0

works great, thanks

And thanks for adding the plugin related data 👍

[schaefi]

There is also a contribution chapter in doc/source/contributing.rst. That information needs to be adapted as it will become obsolete by the changes

[dcermak]

No poetry.lock?

[dcermak]

@Conan-Kudo I was wondering what you think of https://hatch.pypa.io/latest/why/? I have no experience with it, but it appears like a lighter and more upstream aligned build system. WDYT?

[Conan-Kudo]

PyPA disclaims "upstream-alignment" in general, especially after pipx. Mainline Python has been trying to be less opinionated over time about how you build stuff as long as it implements the PEP 517/518 interface.

[Conan-Kudo]

So I think the last thing to figure out is bumpversion...

[schaefi]

All other package builds failed because of

[   40s] RPM build errors:
[   40s]     File not found: /home/abuild/rpmbuild/BUILDROOT/python-kiwi-10.0.0-39.1.x86_64/usr/bin/kiwicompat

we dropped that, it seems to be still in the package list

[Conan-Kudo]

working on those now, and also fixing the PKGBUILD

[schaefi]

btw, why was sphinx added to be a package dependency ? It's imho only a development dependency. That's also why didn't saw this before

[Conan-Kudo]

btw, why was sphinx added to be a package dependency ? It's imho only a development dependency. That's also why didn't saw this before

It's required to build the man pages.

[schaefi]

It's required to build the man pages.

ah so we render the man pages now at package build time. I think before we rendered them with the "make build" target and put them into the tarball. ok your way is better but we have new buildrequires now

[schaefi]

if we don't find a solution we can also render the man pages via rst2man

for man in $(shell find -name "*.rst" | sed -e "[email protected]@@"); do \
        rst2man $$man.rst $$man.8 ;\
    done

I do it that way in another project

[Conan-Kudo]

It's because it's python3-Sphinx in SUSE distributions, easy fix. Please refresh staging.

[schaefi]

ah ok, I will

[schaefi]

done, less unresolvables more errors :)

[   14s] sphinx-build -b man -W -d build/doctrees   source build/man
[   14s] Running Sphinx v7.2.6
[   15s] [autosummary] generating autosummary for: api.rst, api/kiwi.archive.rst, api/kiwi.boot.image.rst, api/kiwi.boot.rst, api/kiwi.bootloader.config.rst, api/kiwi.bootloader.install.rst, api/kiwi.bootloader.rst, api/kiwi.bootloader.template.rst, api/kiwi.builder.rst, api/kiwi.container.rst, ..., working_with_images/disk_setup_for_vagrant.rst, working_with_images/iso_to_usb_stick_deployment.rst, working_with_images/iso_to_usb_stick_file_based_deployment.rst, working_with_images/iso_to_usb_stick_grub2_boot_from_iso.rst, working_with_images/legacy_netboot_root_filesystem.rst, working_with_images/network_live_iso_boot.rst, working_with_images/network_overlay_boot.rst, working_with_images/setup_network_bootserver.rst, working_with_images/setup_yast_on_first_boot.rst, working_with_images/use_suse_media.rst
[   15s] 
[   15s] Warning, treated as error:
[   15s] Failed to import kiwi.system.prepare.
[   15s] Possible hints:
[   15s] * AttributeError: module 'kiwi.system' has no attribute 'prepare'
[   15s] * ModuleNotFoundError: No module named 'pkg_resources'
[   15s] make: *** [Makefile:157: man] Error 2

[schaefi]

I'm out for today. I added you as admin to Virtualization:Appliances:Staging/python-kiwi so you can eat your own dog food

make build
cp dist/* ...
osc ci

[Conan-Kudo]

Everything is now fixed up in https://build.opensuse.org/project/show/Virtualization:Appliances:Staging

[schaefi]

Thanks @Conan-Kudo, I went through a night session to fixup all integration tests due to the shift of some of them into newer distros. Ubuntu tests are based on Lunar now, Debian tests are based on Debian 12, Leap uses backports as the kiwi package there builds for py311.

In a final change here I fixed the python-kiwi-pkgbuild-template to also apply the docopt-ng vs. docopt workaround.
After that all integration tests builds and I will now merge this PR

One issue is the publish to pypi and the publishing of the docs. This we will find out when I try the release, so maybe there will be a hiccup when we release

[glaubitz]

This change added the build dependency pytest_container which neither openSUSE nor Debian or Fedora are actually packaging.

Is kiwi not supposed to be tested during package builds anymore?

[dcermak]

This change added the build dependency pytest_container which neither openSUSE nor Debian or Fedora are actually packaging.

Is kiwi not supposed to be tested during package builds anymore?

pytest_container runs the kiwi integration tests in containers that cannot be run in OBS or koji. You can ignore this test dependency for the purpose of creating an RPM.

[glaubitz]

This change added the build dependency pytest_container which neither openSUSE nor Debian or Fedora are actually packaging.
Is kiwi not supposed to be tested during package builds anymore?

pytest_container runs the kiwi integration tests in containers that cannot be run in OBS or koji. You can ignore this test dependency for the purpose of creating an RPM.

I cannot ignore the dependency:

   dh_auto_test -O--buildsystem=pybuild
I: pybuild base:311: cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12/build; python3.12 -m pytest test
============================= test session starts ==============================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /<<PKGBUILDDIR>>
configfile: setup.cfg
plugins: typeguard-4.3.0, testinfra-10.1.1
collected 1721 items / 1 error

==================================== ERRORS ====================================
__________ ERROR collecting .pybuild/cpython3_3.12/build/test/scripts __________
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
/usr/lib/python3/dist-packages/_pytest/assertion/rewrite.py:184: in exec_module
    exec(co, module.__dict__)
test/scripts/conftest.py:1: in <module>
    from pytest_container import auto_container_parametrize, DerivedContainer
E   ModuleNotFoundError: No module named 'pytest_container'
=========================== short test summary info ============================
ERROR test/scripts - ModuleNotFoundError: No module named 'pytest_container'
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 2.67s ===============================
E: pybuild pybuild:389: test: plugin pyproject failed with: exit code=2: cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12/build; python3.12 -m pytest test
dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p 3.12 returned exit code 13
make: *** [debian/rules:33: binary] Error 25
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

[dcermak]

I guess we should make the dependency optional then. Wdyt @schaefi @Conan-Kudo?

[Conan-Kudo]

That basically means defining a new dependency group. Sure, fine with me.