Tests elf files

[abukharmeh]

Hi,

I wonder if it still make sense to keep tests elf files precompiled in test/riscv-tests ? I think it make more sense to add riscv-arch-test repo as a git submodule of this repository.

Also maybe we should add a new task to CI to compile and run these tests and make sure they all pass on each PR ?

Kind regards,
Ibrahim.

[jrtc27]

Precompiled binaries reduce the barrier to entry for development on the model, as it's quick and easy to run the tests, no need to mess about making sure you have the right toolchain (though LLVM makes that a lot easier). But committing binaries is also not great practice. So it's a trade-off.

Submodules are a bad idea IMO though. The tooling around them isn't great, and can be awkward for those less familiar with git. You also keep all the downsides of vendoring code (that it gets outdated) with about the only benefit being not having extra copies of things bloating repos. I don't think that makes sense as a choice. Instead you probably want a decoupled model of "point sail-riscv at a clone of the test suite and run it", though that relies on said test suite being mature enough to have a stable interface. As for riscv-arch-test specifically, it seems to be overly complicated and poorly designed (every implementation requires its own copy pasta tree of yet more copy pasta between extensions), and I don't understand the future of it when RISCOF is supposedly going to supersede it (which has its own poorly designed way to adapt to different models, keeping some of the copy pasta but instead forcing it out of a centralised repo into each implementation's repo so it's going to get even more out of sync).

[abukharmeh]

I opened this issue because the test files in this repo are out of date, and that need to addressed somehow.

I suggested submodules because we can just track the arch test repo, and use all tests they add for various new extension without a lot of hassle. If precompiled bins are a must, then maybe GH actions can be used to compile them either on this repo, or arch-test one.