rez-cook: a rez package manager

[anderslanglands]

Hi there! My weekend project for the last few weeks has been a package build/dependency manager built as a wrapper around rez-build, which for now I’m calling rez-cook. You can find the repo here, and the recipe repo here. 

I built this because I'm fed up of building the same set of packages every time I need to set up a new os, and then trying to remember the right combination of dependencies and magical cmake incantations to get everything playing nicely together. I mentioned this to Allan and he suggested it might serve as a useful example for something like this in rez itself.

Given a package request to cook, it will find a matching recipe, and the recipes for its entire dependency tree, figure out which ones need to be built (and which variant to build), and then build and install them in reverse dependency order. 

You can also specify constraints, so for instance:

rez-python ./rez-cook.py usd --constrain vfxrp-2022 cfg-release

Will build the latest version of usd that there is a recipe for. --constrain specifies the list of constraints (just PackageRequests for which there is a matching recipe) and causes the dependency resolution to match the dependencies of usd to vfxrp-2022 which is a recipe that defines the list of dependencies for vfx reference platform 2022.

cfg-release in the above is a required constraint specifying whether to build in debug or release mode, and basically just maps to a an "any"-versioned part of the variant in the recipe. Not specifying any of these (and not having them defined implicitly by the dependency resolution) will abort the cook and prompt the user to constrain them properly. 

It will also install Python dependencies from PyPI. These are handled similarly to the way rez-pip does it but I had to reimplement this to get the dependency resolution to work with the rest of the system. 

Other packages are either built from source (either from a release archive or by cloning a repo) using rez-build, or in some cases by downloading binaries and installing them where building from source doesn't add value (e.g. cmake). Source builds also support patching, mostly for fixing up broken cmake (ugh).

Limitations:

• rez-python must be python-3.6+. Trying to support py2 as well was too much headache.
• the dependency resolution (as well as a bunch of utility classes) is home-cooked and probably crappy. I don’t know the rez codebase well and it was simpler for me to do it this way. I've likely reimplemented a bunch of functionality all over the place that would be better off just using existing modules in rez.
• Similarly, I'm injecting a few functions into the recipe package.py's for doing things like downloading archives, cloning repos etc. These would likely be better off as utility modules one could import, but this was the quickest way for me to do it at the time.
• it works on my machines. I’ve tested on Windows 11 and Ubuntu 20. Doubtless contains lots of bugs that will explode in other configurations. 
• only pwsh.exe is supported on windows. because cmd.exe is awful and supporting both pwsh and sane posix shells was enough work already.

For the most part it’s now at a stage where it does what I need: building the typical VFX software stack for my own development purposes at home. There’s a bunch of stuff that would make it better though:

1. Features - Similar to how cargo works maybe, recipes should be able to define optional features users can turn on at cook time that may or may not imply different sets of dependencies. 
2. Run-time variant overrides - This is more of a rez thing. It would be great to be able to override the resolved variant of a particular package or set of packages for inserting debug versions of only some packages. (This only applies to Linux because Windows is awful and breaks the c++ ABI in debug builds so basically the entire tree needs to be debug anyway.) 
3. macOS support - I don’t use mac these days so can’t really do much here. 
4. apt/yum support - Some libraries need a bunch of things like xorg-dev on Linux that I really don’t want to make recipes for. I don’t think we want rez to be running apt install but checking what's installed against a list of required packages and prompting the user to do it would be helpful. 
5. Download caching - Currently cooking a recipe always downloads the source archive. Would be nice to cache these and not download again when building multiple variants of the same source code. 
6. More packages - Obviously. There's also the rather tedious work of figuring out exactly what combinations of dependencies work to specify the requires of each recipe as loosely as possible.
7. Optimization - Building the dep tree for usd can be pretty slow if there are lots of packages installed.

[fnaum]

Hi Andres, That is pretty neat, thank you for sharing. I took a quick look at the code and recipes. I'm pretty sure it would be of great value to many studios adopting rez, as they will be saving a great deal of time with all these recipes. I can see something like the pre_cook, patch could make it to rez, that way people that are not too familiar with CMake can bundle the build recipe inside the package as you do there. I have a question about your motivation behind this, when you say "I built this because I'm fed up of building the same set of packages every time I need to set up a new os" is that because you are moving to different studios? or you meant that you have to get all the new required version of the dependencies of the package. If I got it right, when you want to get a new version of usd, you just copy the recipe from the previous version, and update the version of the dependencies and then by running: rez-python ./rez-cook.py usd --constrain vfxrp-2022 cfg-release all the dependencies will be downloaded and built in reverse dependency order, without the need (most of the time) to update the recipes of the dependencies, right? or do you still need to go thru each dependency and create a new folder with the new version and recipe before attempting the above command? Cheers, Fede

…

On Wed, Jul 13, 2022 at 5:19 PM Anders Langlands ***@***.***> wrote: Hi there! My weekend project for the last few weeks has been a package build/dependency manager built as a wrapper around rez-build, which for now I’m calling rez-cook. You can find the repo here <https://github.com/anderslanglands/rez-cook>, and the recipe repo here <https://github.com/anderslanglands/rez-recipes>. I built this because I'm fed up of building the same set of packages every time I need to set up a new os, and then trying to remember the right combination of dependencies and magical cmake incantations to get everything playing nicely together. I mentioned this to Allan and he suggested it might serve as a useful example for something like this in rez itself. Given a package request to cook, it will find a matching recipe, and the recipes for its entire dependency tree, figure out which ones need to be built (and which variant to build), and then build and install them in reverse dependency order. You can also specify constraints, so for instance: rez-python ./rez-cook.py usd --constrain vfxrp-2022 cfg-release Will build the latest version of usd that there is a recipe for. --constrain specifies the list of constraints (just PackageRequests for which there is a matching recipe) and causes the dependency resolution to match the dependencies of usd to vfxrp-2022 which is a recipe that defines the list of dependencies for vfx reference platform 2022. cfg-release in the above is a required constraint specifying whether to build in debug or release mode, and basically just maps to a an "any"-versioned part of the variant in the recipe. Not specifying any of these (and not having them defined implicitly by the dependency resolution) will abort the cook and prompt the user to constrain them properly. It will also install Python dependencies from PyPI. These are handled similarly to the way rez-pip does it but I had to reimplement this to get the dependency resolution to work with the rest of the system. Other packages are either built from source (either from a release archive or by cloning a repo) using rez-build, or in some cases by downloading binaries and installing them where building from source doesn't add value (e.g. cmake). Source builds also support patching, mostly for fixing up broken cmake (ugh). Limitations: - rez-python must be python-3.6+. Trying to support py2 as well was too much headache. - the dependency resolution (as well as a bunch of utility classes) is home-cooked and probably crappy. I don’t know the rez codebase well and it was simpler for me to do it this way. I've likely reimplemented a bunch of functionality all over the place that would be better off just using existing modules in rez. - Similarly, I'm injecting a few functions into the recipe package.py's for doing things like downloading archives, cloning repos etc. These would likely be better off as utility modules one could import, but this was the quickest way for me to do it at the time. - it works on my machines. I’ve tested on Windows 11 and Ubuntu 20. Doubtless contains lots of bugs that will explode in other configurations. - only pwsh.exe is supported on windows. because cmd.exe is awful and supporting both pwsh and sane posix shells was enough work already. For the most part it’s now at a stage where it does what I need: building the typical VFX software stack for my own development purposes at home. There’s a bunch of stuff that would make it better though: 1. *Features* - Similar to how cargo works maybe, recipes should be able to define optional features users can turn on at cook time that may or may not imply different sets of dependencies. 2. *Run-time variant overrides* - This is more of a rez thing. It would be great to be able to override the resolved variant of a particular package or set of packages for inserting debug versions of only some packages. (This only applies to Linux because Windows is awful and breaks the c++ ABI in debug builds so basically the entire tree needs to be debug anyway.) 3. *macOS support* - I don’t use mac these days so can’t really do much here. 4. *apt/yum support* - Some libraries need a bunch of things like xorg-dev on Linux that I really don’t want to make recipes for. I don’t think we want rez to be running apt install but checking what's installed against a list of required packages and prompting the user to do it would be helpful. 5. *Download caching* - Currently cooking a recipe always downloads the source archive. Would be nice to cache these and not download again when building multiple variants of the same source code. 6. *More packages* - Obviously. There's also the rather tedious work of figuring out exactly what combinations of dependencies work to specify the requires of each recipe as loosely as possible. 7. *Optimization* - Building the dep tree for usd can be pretty slow if there are lots of packages installed. — Reply to this email directly, view it on GitHub <#1347>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVRQRFGFXJIT3EXEAPLRILVTZUY5ANCNFSM53NUTREA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[anderslanglands]

Hi Federico, Regarding motivation, my requirements are just for home use (though once it matures I would use it at work as well). I do a lot of development at home, so need the whole vfx software stack. And it seems like about once a year on average I need to set everything up from scratch again (either on a new machine, VM, wiped install etc.). This is often enough that it’s annoying, and infrequent enough that I never remember all the gotchas for each package. Regarding your last question: yes if the new version of a recipe builds with the same dependencies as an existing one, you can just copy the existing one and update its version. You only need to create new dependency recipes for it if the new version now relies on a dependency you don’t have a recipe for. Cheers, Anders. On Thu, 14 Jul 2022 at 01:42, Federico Naum ***@***.***> wrote:

…

Hi Andres, That is pretty neat, thank you for sharing. I took a quick look at the code and recipes. I'm pretty sure it would be of great value to many studios adopting rez, as they will be saving a great deal of time with all these recipes. I can see something like the pre_cook, patch could make it to rez, that way people that are not too familiar with CMake can bundle the build recipe inside the package as you do there. I have a question about your motivation behind this, when you say "I built this because I'm fed up of building the same set of packages every time I need to set up a new os" is that because you are moving to different studios? or you meant that you have to get all the new required version of the dependencies of the package. If I got it right, when you want to get a new version of usd, you just copy the recipe from the previous version, and update the version of the dependencies and then by running: rez-python ./rez-cook.py usd --constrain vfxrp-2022 cfg-release all the dependencies will be downloaded and built in reverse dependency order, without the need (most of the time) to update the recipes of the dependencies, right? or do you still need to go thru each dependency and create a new folder with the new version and recipe before attempting the above command? Cheers, Fede On Wed, Jul 13, 2022 at 5:19 PM Anders Langlands ***@***.***> wrote: > Hi there! My weekend project for the last few weeks has been a package > build/dependency manager built as a wrapper around rez-build, which for now > I’m calling rez-cook. You can find the repo here > <https://github.com/anderslanglands/rez-cook>, and the recipe repo here > <https://github.com/anderslanglands/rez-recipes>. > > I built this because I'm fed up of building the same set of packages every > time I need to set up a new os, and then trying to remember the right > combination of dependencies and magical cmake incantations to get > everything playing nicely together. I mentioned this to Allan and he > suggested it might serve as a useful example for something like this in rez > itself. > > Given a package request to cook, it will find a matching recipe, and the > recipes for its entire dependency tree, figure out which ones need to be > built (and which variant to build), and then build and install them in > reverse dependency order. > > You can also specify constraints, so for instance: > > rez-python ./rez-cook.py usd --constrain vfxrp-2022 cfg-release > > Will build the latest version of usd that there is a recipe for. > --constrain specifies the list of constraints (just PackageRequests for > which there is a matching recipe) and causes the dependency resolution to > match the dependencies of usd to vfxrp-2022 which is a recipe that > defines the list of dependencies for vfx reference platform 2022. > > cfg-release in the above is a required constraint specifying whether to > build in debug or release mode, and basically just maps to a an > "any"-versioned part of the variant in the recipe. Not specifying any of > these (and not having them defined implicitly by the dependency resolution) > will abort the cook and prompt the user to constrain them properly. > > It will also install Python dependencies from PyPI. These are handled > similarly to the way rez-pip does it but I had to reimplement this to get > the dependency resolution to work with the rest of the system. > > Other packages are either built from source (either from a release archive > or by cloning a repo) using rez-build, or in some cases by downloading > binaries and installing them where building from source doesn't add value > (e.g. cmake). Source builds also support patching, mostly for fixing up > broken cmake (ugh). > > Limitations: > > - rez-python must be python-3.6+. Trying to support py2 as well was > too much headache. > - the dependency resolution (as well as a bunch of utility classes) is > home-cooked and probably crappy. I don’t know the rez codebase well and it > was simpler for me to do it this way. I've likely reimplemented a bunch of > functionality all over the place that would be better off just using > existing modules in rez. > - Similarly, I'm injecting a few functions into the recipe > package.py's for doing things like downloading archives, cloning repos etc. > These would likely be better off as utility modules one could import, but > this was the quickest way for me to do it at the time. > - it works on my machines. I’ve tested on Windows 11 and Ubuntu 20. > Doubtless contains lots of bugs that will explode in other configurations. > - only pwsh.exe is supported on windows. because cmd.exe is awful and > supporting both pwsh and sane posix shells was enough work already. > > For the most part it’s now at a stage where it does what I need: building > the typical VFX software stack for my own development purposes at home. > There’s a bunch of stuff that would make it better though: > > 1. *Features* - Similar to how cargo works maybe, recipes should be > able to define optional features users can turn on at cook time that may or > may not imply different sets of dependencies. > 2. *Run-time variant overrides* - This is more of a rez thing. It > would be great to be able to override the resolved variant of a particular > package or set of packages for inserting debug versions of only some > packages. (This only applies to Linux because Windows is awful and breaks > the c++ ABI in debug builds so basically the entire tree needs to be debug > anyway.) > 3. *macOS support* - I don’t use mac these days so can’t really do > much here. > 4. *apt/yum support* - Some libraries need a bunch of things like > xorg-dev on Linux that I really don’t want to make recipes for. I don’t > think we want rez to be running apt install but checking what's > installed against a list of required packages and prompting the user to do > it would be helpful. > 5. *Download caching* - Currently cooking a recipe always downloads > the source archive. Would be nice to cache these and not download again > when building multiple variants of the same source code. > 6. *More packages* - Obviously. There's also the rather tedious work > of figuring out exactly what combinations of dependencies work to specify > the requires of each recipe as loosely as possible. > 7. *Optimization* - Building the dep tree for usd can be pretty slow > if there are lots of packages installed. > > — > Reply to this email directly, view it on GitHub > <#1347>, or > unsubscribe > < https://github.com/notifications/unsubscribe-auth/ABVRQRFGFXJIT3EXEAPLRILVTZUY5ANCNFSM53NUTREA > > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> > — Reply to this email directly, view it on GitHub <#1347 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOYQXPG5VPZ2XR2CCKSRV3VT3BTRANCNFSM53NUTREA> . You are receiving this because you authored the thread.Message ID: <AcademySoftwareFoundation/rez/repo-discussions/1347/comments/3137794@ github.com>

[fnaum]

Thank you @anderslanglands, I can see repeating all this once a year is a good reason for you to create this.
Most of the time I need some of the AL in-house packages to do some development and our rez ecosystem is too big to be able to replicate it on a home computer, but I can see many things in this project we can either think about moving to rez.

Thanks for the contribution