feat: add hyperscan support

[ffontaine]

hyperscan will run simultaneously all version checkers on a file which reduce processing time.

pyperscan package is used instead of the most well-known hyperscan package as pyperscan allows to add a tag for each pattern. This feature will allow to retrieve easily the checker associated to the matched pattern.

Fix #2485

Signed-off-by: Fabrice Fontaine [email protected]

[terriko]

This is so cool, but it looks like it's not getting the same scan results in 28 tests. Here's a snippet:

FAILED test/test_scanner.py::TestScanner::test_version_in_package[http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/-rh-nodejs14-nodejs-14.16.0-1.el7.x86_64.rpm-node.js-14.16.0-other_products585] - AssertionError: assert '14.16.0' in {'1.0.2k', '1.2.11', '10.12.0'}
FAILED test/test_scanner.py::TestScanner::test_version_in_package[https://kojipkgs.fedoraproject.org/packages/openssh/6.8p1/1.1.fc23/x86_64/-openssh-clients-6.8p1-1.1.fc23.x86_64.rpm-openssh-6.8p1-other_products608] - AssertionError: assert '6.8p1' in {'3.1'}
FAILED test/test_scanner.py::TestScanner::test_version_in_package[https://www.rpmfind.net/linux/fedora/linux/development/rawhide/Everything/aarch64/os/Packages/t/-thunderbird-102.5.1-1.fc38.aarch64.rpm-thunderbird-102.5.1-other_products787] - AssertionError: sqlite not found in thunderbird-102.5.1-1.fc38.aarch64.rpm. Remove sqlite from other_products.
assert 'sqlite' in {'libjpeg', 'libjpeg-turbo', 'lz4', 'rust', 'thunderbird'}
FAILED test/test_scanner.py::TestScanner::test_version_in_package[https://downloads.openwrt.org/releases/packages-19.07/x86_64/packages/-tor_0.4.5.10-1_x86_64.ipk-tor-0.4.5.10-other_products799] - AssertionError: assert '0.4.5.10' in {'0.1.2.17', '1.1.1n'}
FAILED test/test_scanner.py::TestScanner::test_version_in_package[http://rpmfind.net/linux/epel/9/Everything/aarch64/Packages/z/-zeek-core-4.2.0-1.el9.aarch64.rpm-zeek-4.2.0-other_products883] - AssertionError: sqlite not found in zeek-core-4.2.0-1.el9.aarch64.rpm. Remove sqlite from other_products.
assert 'sqlite' in {'zeek'}
==== 28 failed, 1513 passed, 43 skipped, 65 warnings in 1656.27s (0:27:36) =====

It also looks like it's not installing on windows because the wheel won't build:

  ERROR: Failed building wheel for pyperscan
ERROR: Could not build wheels for pyperscan, which is required to install pyproject.toml-based projects
Successfully built cve-bin-tool gsutil
Failed to build pyperscan

I didn't dig through the actual error messages deeply but we could be missing a component for the build or something.

[ffontaine]

Indeed, I'll fix those build failures in separate PR (for example, Tor signature can raise false positives)

[ffontaine]

#2681, #2682, #2684, #2688, #2689, #2690 and #2691 should be merged then I'll rebase this PR to have a clean hyperscan addition

[terriko]

The relevant PRs should be merged now, whenever you've got time to get back to this one.

[ffontaine]

I updated the PR but the tests don't start

[ffontaine]

Still working on this one, I assume that we should wait EOL of python 3.7 before merging

[terriko]

We could also make it an optional addition, similar to what we do with PDF export. I don't know off the top of my head if an option can require a different version of python the same way it requires extra packages, but we could just document the heck out of it until 3.7 support gets dropped officially.