Dev/test dependecies in install_requires clash with dev/test dependencies of other packages

[astafan8]

Hi! In a very convoluted way I've discovered this very interesting and amazing package, and I faced the following issue: the test/dev dependencies are part of the runtime dependencies of the forge package, here:
https://github.com/forgepackages/forge/blob/e2692d1be7a07c698dfbcbb2c42009cad2bbdf80/pyproject.toml#L47-L56

as a result, when i install the forge package, as a dependency of my package, the versions of black/pytest/etc start clashing with the versions of black/pytest that are used in my package. In other words, if I first install black/pytest/etc that I'd like to have for my package, and then install the forge package, i'll get the versions of black/pytest/etc that are pinned runtime dependencies of the forge. This effectively prevents me from using black/pytest/etc that I'd like to have in my package (well, it does not prevent, but i have to be mindful of the order of installations, etc).

Could you explain why that is? Perhaps it is part of being "opinionated way of implementing Django apps"? :) I am also veeeeery new to the forge package, so I might be missing a context of how it is used, so I apologize in advance if that's the case.

The recommendation that I found constructive and wide-spread is to separate dev/test dependencies like black/pytest/mypy/etc from the runtime dependencies of the package. Not only it enables the dependent packages define their own dev/test dependencies as they'd like, but also removes the amount of runtime dependencies of a package which speeds up its installation and simplifies maintenance.

[davegaeddert]

Hey @astafan8 — thanks for asking! I'd be curious what the "very convoluted" discovery was 😆

You're right that the way these dependencies are set up is less than ideal. The reason they aren't dev dependencies is because then when you install forge as a dependency, you wouldn't get forge's dev dependencies (so we'd have to also ask you to manually add pytest, black, etc.)

I'm not aware of any major downsides to having these at runtime, but you're right it could/would speed up the deployment installation process at the very least. And obviously if you're having a conflict, that's a problem too! Can you say more about which versions are conflicting? My initial thought would be to just "loosen" the ranges Forge asks for (so maybe pytest = ">=6.0.0" or something instead of pytest = "^7.0.0").

Beyond that, I think properly using dev dependencies would involve changing up the structure so the end user doesn't just install forge, but individually install the various forge-X packages, putting some in runtime and some in dev dependencies.

For today though I think we could definitely loosen up the constraints on some of these:
https://github.com/forgepackages/forge/blob/e2692d1be7a07c698dfbcbb2c42009cad2bbdf80/pyproject.toml#L23-L55

[davegaeddert]

Hey @astafan8 -- long story short, I decided to go ahead and switch up the structure of Forge (other conversations have been pointing go this too). The forge package itself is essentially deprecated, and instead the focus is going to be on selecting forge-x packages individually and making sure that individually they're all compatible with existing Django projects. In practice a pyproject.toml will look more like this (where you can decide what goes in dev dependencies):
https://github.com/forgepackages/starter-template/blob/master/pyproject.toml

So this is the current list of packages:

• forge-work - All-in-one local dev command
• forge-tailwind - Tailwind CSS without JavaScript
• forge-heroku - Streamlined Heroku deploy
• forge-importmap - Use JS libraries with import maps
• forge-test - Pytest + Coverage for Django
• forge-precommit - a git pre-commit hook
• forge-db - Local Postgres db via Docker
• forge-logging - Django logging utils
• forge-format - Unified formatter for Django (uses Black and Ruff)
• forge-htmx - An HTMX integration with template fragments and view actions
• forge-querystats (Pro) - On-page db query info in dev and production
• forge-requestlog (Pro) - Local request/response history with replay
• forge-impersonate (Pro) - Impersonate users in your app
• forge-stafftoolbar (Pro) - UI for staff members
• forge-oauth (Pro) - Minimalist OAuth for Django
• forge-stripe (Pro) - Integrate Stripe with Django models
• forge-sentry (Pro) - Integrate Sentry with backend and frontend
• forge-googleanalytics (Pro) - Google Analytics 4 for frontend and backend events

I need to write docs yet for some, but if you let me know what exactly you're interested in, we can make sure it works well! Thanks for the feedback.

[astafan8]

when you install forge as a dependency, you wouldn't get forge's dev dependencies (so we'd have to also ask you to manually add pytest, black, etc.)

ah, i understand, the idea is to give users all the packages they need once forge is installed, so that forge format/etc wroks, and since forge is opinionated then users just have to follow what's provided :)

Beyond that, I think properly using dev dependencies would involve changing up the structure so the end user doesn't just install forge, but individually install the various forge-X packages, putting some in runtime and some in dev dependencies

another alternative could be to have pip install forge[tests,format] where tests and format are part of .extras_require in setup.cfg-file terms. But i also see why this is not going to work so smoothly for the forge's use case.

Can you say more about which versions are conflicting? My initial thought would be to just "loosen" the ranges Forge asks for (so maybe pytest = ">=6.0.0" or something instead of pytest = "^7.0.0"

yes, loosening things up to define a minimal version or perhaps indeed specifying only the major version that is supposed to be used would give more flexibility. however, this is not such an issue for pytest as it is for black - because with minor version bumps there could be formatting changes. and of course it is important to lock the version-of-black-used in a lock file (e.g. requirements.txt, or poetry.lock if poetry is used)

---

The forge package itself is essentially deprecated, and instead the focus is going to be on selecting forge-x packages individually and making sure that individually they're all compatible with existing Django projects

oh, that change came very fast! :) this looks good, makes a lot of sense!

I need to write docs yet for some, but if you let me know what exactly you're interested in, we can make sure it works well!

thanks! i don't have any specific interest yet, just looking around at the moment,

Thanks for the feedback.

you're very welcome! :)

[davegaeddert]

Sorry I sort of repeated what you said there — the reason we include pytest and black (for example) in Forge and want downstream users to get them is because of the commands forge test, forge pre-commit, and forge format. For the most part, the expectation for the literal forge package is that people are starting from the Forge Quickstart (https://www.forgepackages.com/docs/forge/quickstart/) for a new Django project, not integrating all of this into an existing project. Does that help?