Build system ideas

[nopeslide]

Just thinking out loud regarding our build system.

onnx is huge and I think our build system can not scale good enough, so we need to address a few things:

1. connxr is currently completely dependant on the onnx_generator

• makes customizations hard, since they would have to be in sync with the onnx_generator
• onnx_generator should act as template builder, not as hacky configuration tool

2. If we add types to our already large set of settings (all valid combinations of onnx version, operator version,onnx domain, operator) options will explode, did not explode until now, because of our limited number of implementations

• almost all options are needed (at least somewhere)
• central large config would be nice
  • no parameter search anymore, there is a documented setting for it
  • builds are consistent and reproducible
• options have relations
  • i.e. if we disable a domain globally, it must disable all related operators
• options must be evaluated by the preprocessor

3. we have python scripts

• we should have a venv for these

ideas

• use Kconfig as config generator
  • see python Kconfiglib
  • we would have "graphical" hierarchical menus of documented options
  • Kconfig can include itself
    • hierarchical menus can correspond to directories (domain, operators, version)
  • generates a single file with all options that can be included by make
  • make is configured with the config
    • make can handle the build process
  • preprocessor is configured with the config
    • preprocessor can handle code configurations
• generate operator set with the preprocessor instead of onnx_generator
  • onnx_generator does not control the build process anymore
  • utilize Kconfigs configuration options to filter list elements inside the operator set
  • utilize __COUNTER__ macro to count included elements and set the list length right
    • or maybe use Kconfig or make to determine list length?
• use hierarchical Makefiles
  • produce archives or partially linked object files
  • Makefiles are simpler, easier to maintain

[nopeslide]

@robinvanemden come out of the shadows, you emoji guerrilla.
what's your interest in connxr? :D

[robinvanemden]

Hi @nopeslide! I am currently finishing another project, which is keeping me rather busy (ergo the emoji). But for my next project, I need to convert ONNX (including the ML ops) to C99. So if cONNXr didn't exist, I would have had to invent it 😃 I hope to be able to contribute more actively soon!

[nopeslide]

Hi @nopeslide! I am currently finishing another project, which is keeping me rather busy (ergo the emoji). But for my next project, I need to convert ONNX (including the ML ops) to C99. So if cONNXr didn't exist, I would have had to invent it smiley I hope to be able to contribute more actively soon!

funny story, it's the same reason I started working on connxr. I need a simple framework to integrate it into simulations, especially the ML operators. I also had the options to start from scratch, but found connxr while going through all github onnx projects :D
that's one of the reasons I push it in the direction of being easy to customize and extend. look forward to your contributions ;D

[alrevuelta]

Great that you find it useful :D

connxr is currently completely dependant on the onnx_generator
makes customizations hard, since they would have to be in sync with the onnx_generator
onnx_generator should act as template builder, not as hacky configuration tool

We talked a bit about the role of onnx_generator in #50. My view of onnx_generator is that it should be totally dependant on onnx. So we target one specific onnx version (ideally the latest, and we keep evolving as onnx does) so we have always what is latest (operator sets versions and schema). So I would keep it as it is, so it just calls the script and generates all operators/resolvers/... of the latest onnx version.

Then, on the other side we can have the building system that you are referring to. In my opinion this building system should take the following inputs:

• Operator set to support, i.e. 10.
• Specific operators to support. If someone doesn't want to have all 123123143423 operators in the binary, just compile and link the ones that are relevant.
• Specific data type. Maybe someone just wants the float type, so not all functions should be compiled.

All previous inputs are just an idea, but I think it would be good to think about them. Maybe some more are needed.

Regarding KConfig, nothing against and I have never used. If we can benefit from it of course you can use it, but can't we use some other kind of tool? Don't think that we need this graphical UI menus. Maybe some kind of configurable Makefile would be sufficient. Or Python generating a Makefile? Haven't used CMake much, could it be useful here?

onnx_generator does not control the build process anymore

Is it controlling it? Its just calling the Python script that generates the files/funcions. What do you mean by controlling the build process?

we should have a venv for these

How do you think we can do this? Shouldn't the venv be responsibility of the user that is using the software. Isn't it enough having the requirements file?

[alrevuelta]

One more comment regarding onnx_generator. I see this as some kind of way of updating our repo to the latest onnx version. Hei! onnx released a new version. Ok, lets run onnx_generator to generate the newest operators and implement them.

And btw, I plan to include in onnx_generator also a script to pull all the testcases for all operators. Found a way to walk through all previous versions and have all test cases. So whenever a new version is released, we can get also the test cases for that new opset. Note that onnx testing backend only contains the latest opset version as discussed onnx/onnx#2912

[nopeslide]

We talked a bit about the role of onnx_generator in #50. My view of onnx_generator is that it should be totally dependant on onnx. So we target one specific onnx version (ideally the latest, and we keep evolving as onnx does) so we have always what is latest (operator sets versions and schema). So I would keep it as it is, so it just calls the script and generates all operators/resolvers/... of the latest onnx version.

I'm with you, onnx_generator should only use a single onnx version as reference and nothing more.
I'm against generating everything in one go, because it will then update/overwrite everything.
I want one tool to generate templates for a specific operator, and one tool to generate global structures like onnx sets.
So the sets are sth we only touch when onnx updates, and operators can be added as we go.

Then, on the other side we can have the building system that you are referring to. In my opinion this building system should take the following inputs:

* Operator set to support, i.e. 10.

* Specific operators to support. If someone doesn't want to have all 123123143423 operators in the binary, just compile and link the ones that are relevant.

* Specific data type. Maybe someone just wants the `float` type, so not all functions should be compiled.

All previous inputs are just an idea, but I think it would be good to think about them. Maybe some more are needed.

Yes at least these options, I already have a demo, will continue working on it when #57 is through.

Regarding KConfig, nothing against and I have never used. If we can benefit from it of course you can use it, but can't we use some other kind of tool? Don't think that we need this graphical UI menus. Maybe some kind of configurable Makefile would be sufficient. Or Python generating a Makefile? Haven't used CMake much, could it be useful here?

CMake is hell imo. It is essentially make, but overloaded with so many macros, either sth happen automagically or you are searching hours inside its documentation. make on the other hand is plain stupid and you have to do a lot of stuff manually, but you can just do it.

KConfig generates a list of defines for make and the preprocessor. It handles the config and puts out these two files to be integrated in the normal workflow. the Graphical ui is optional, but appreciated at some point (it show you dependencies between options and help messages).
The idea behind Kconfig is simple: you write your normal Makefile (but with Kconfig compatible variables) and the Kconfig generated variable definitions just overwrite your default make options.

onnx_generator does not control the build process anymore

Is it controlling it? Its just calling the Python script that generates the files/funcions. What do you mean by controlling the build process?

Since we need it to generate the operator sets, it is controlling the build. You can't configure the operator set, you have to generate it. Would love to have this configuration inside the actual build system (since it's a build configuration) instead of replacing code.

we should have a venv for these

How do you think we can do this? Shouldn't the venv be responsibility of the user that is using the software. Isn't it enough having the requirements file?

I would use our requirements.txt to build a venv via make. the use can still choose the construct their own venv, but we should at least generate our own as default.