Double the speed of zfec

[itamarst]

Fixes #102

• Twice as fast.
• No longer supports 3.8 (possibly not actually necessary, I was thinking there was manylinux issues but could be wrong. In practice 3.8 is EOL anyway for security updates, so meh).
• Won't install on Python 3.9.0-3.9.5 unless they update pip in a virtualenv... in which case pip will just install the previous release, so not actually a problem for users.
• No longer supports Ubuntu 18.04 (but it's EOL too).

[meejah]

🎉

[hacklschorsch]

I have machines older than this ; what's the best way to run tahoe-lafs on them?

[meejah]

Can you be more specific about the machines in question?

(The generic answer would be "use a ZFEC library before version 1.6.0.0").

[hacklschorsch]

My home server is an Intel Atom second generation or so, it is x86-64-v1. This also broke two Tahoe-LAFS CI runners for Redhat-Based distros - @meejah says because their CPUs are too old - are you sure about your 2008/2009 estimation? It would be great if there was a documented way of running zfec on older hardware. What are the steps to build this myself without this optimization? Can I do it without editing the source?

[hacklschorsch]

I guess something like hwcaps checking is too much hassle? https://www.theregister.com/2022/12/16/tumbleweed_reverses_x864v2_plan/

The hwcaps feature in glibc allows detection and manipulations of the hardware capabilities of chips in various CPU families

[hacklschorsch]

https://pypi.org/project/mwa-hyperbeam/ say they offer multiple wheels targeting different microarchitecture levels:

What are these different x86-64 versions?

They are microarchitecture levels. By default, Rust compiles for all x86-64 CPUs; this allows maximum compatibility, but potentially limits the runtime performance because many modern CPU features can't be used. Compiling at different levels allows the code to be optimised for different classes of CPUs so users can get something that works best for them.

Looking at the download files for their latest version I don't see how they do it though

[itamarst]

There are multiple ways to handle different CPU microarchitectures:

1. Use oldest possible CPU target; this is the default.
2. Use a commonly-used CPU target, what we tried to do in this PR, which I guess is a problem.
3. Runtime dispatch inside the function, which is supported by C/C++/Rust and probably other languages, where you (a) compile multiple versions of a function and (b) choose one at runtime. I imagine this is annoying in C.
4. Import time! This is a fun Python-specific one, where you ship multiple copies of the extension compiled against different targets, and then at import time check in Python what CPU you have, and then import the appropriate one for the current CPU. I have prototyped this.

For zfec... maybe easier just to revert this change for now?

[hacklschorsch]

I am all for execution efficiancy and would be sad to see this reverted. "Import time" seems like a great solution to this - can I help with realizing it? Maybe with code review and/or testing?