Release v1.5.2

[msm-cert]

This release consists almost exclusively of performance improvements. Query speed is reduced by over 50% in the tested workloads.

Features:

• Add a verbose mode (-v) that also prints debug logs (Add debug logs, including query plans #218)

Performance:

• Refactor (Add debug logs, including query plans #218)
• Add debug logs, including query plans (Add debug logs, including query plans #218)
• opt1: Flatten trivial operations (opt1: flatten_trivial_operations #220)
• opt2: Inline suboperations (opt2: inline_suboperations #221)
• opt3: Deduplicate primitives (opt3: deduplicate_primitives #222)
• opt4: Simplify minof in some cases (opt4: simplify_minof #223)
• opt5: Propagate degenerate queries (opt5: propagate_degenerate_queries #224)
• opt6: Reorder subqueries more optimally (opt6: reorder_subqueries #225)
• opt7: Prefetch disk reads (opt7: prefetching #226)
• opt8: Keep runs in compressed form for longer (opt8: keep runs in compressed form #227)

Correctness:

Refactoring and maintenance:

• Create a simple framework for query optimizations (Refactor primitive queries into AND of atoms #217)

---

Tests were performed on a dataset consisting of 9416097 files (over 2TB of data), using a single server with a HDD RAID, and by matching all yara rules from https://github.com/Neo23x0/signature-base/ (that were not "degenerate", i.e. broken). No user-visible changes other than performance were discovered (i.e. still the same files returned). The measured time is a sum of time taken by all matching.

Name	Time
opt0_1ffe9c01 --> opt1_1ffe9c01	3648s --> 3548s (-100, -2.72%)
opt1_1ffe9c01 --> opt2_1ffe9c01	3548s --> 3583s (+34, +0.98%)
opt2_1ffe9c01 --> opt3_1ffe9c01	3583s --> 3341s (-243, -6.76%)
opt3_1ffe9c01 --> opt4_1ffe9c01	3341s --> 3164s (-178, -5.30%)
opt4_1ffe9c01 --> opt5_1ffe9c01	3164s --> 3280s (+116, +3.68%)
opt5_1ffe9c01 --> opt6_1ffe9c01	3280s --> 3003s (-278, -8.45%)
opt6_1ffe9c01 --> opt7_1ffe9c01	3003s --> 2415s (-589, -19.58%)
opt7_1ffe9c01 --> opt8_1ffe9c01	2415s --> 1635s (-781, -32.30%)
opt8_1ffe9c01 --> opt9_1ffe9c01	1635s --> 1378s (-257, -15.70%)
total improvement	3648s --> 1378s (-2270, -62.22%)

[msm-cert]

Just to be clear, for mquery users: this only speeds up the first part of the matching (prefiltering). The second part of mquery work - yara matching - is unaffected by these changes, and in most cases is expected to be slower.

But it does make mquery faster, just not by 250% :).