Would you accept a convenience function for replace_all that builds the AhoCorasick automaton for you?

[arifd]

Hi Andrew!

replace_all requires two separate lists of equal length. These means the function may panic if you pass in differing array lengths.

But for my interest, more importantly, It's hard to line up the haystacks, in code, with the replace_withs because they are on separate lines.

Both these problems can be eliminated with this utility fn I made:

const fn aho_corasick_builder<'a, const N: usize>(
    tuples: [(&'a str, &'a str); N],
) -> ([&'a str; N], [&'a str; N]) {
    let mut lhs = [""; N];
    let mut rhs = [""; N];
    let mut i = 0;

    while i < N {
        lhs[i] = tuples[i].0;
        rhs[i] = tuples[i].1;
        i += 1;
    }

    (lhs, rhs)
}

So now I can write code that looks like this:

const REPLACER: ([&str; 19], [&str; 19]) = aho_corasick_builder([
    ("^", "\\^"),       // Boost: Used to indicate boosting in query relevance scoring.
    (":", "\\:"),       // Field: Used to separate field names and values.
    (" TO ", " \\TO "), // Range: Indicates a range query (e.g., date ranges).
    ("[", "\\["),       // Range: Used to denote inclusive range queries.
    ("]", "\\]"),       // Range: Used to denote inclusive range queries.
    ("{", "\\{"),       // Range: Used to denote exclusive range queries.
    ("}", "\\}"),       // Range: Used to denote exclusive range queries.
    ("<", "\\<"),       // Range: Indicates less-than range.
    (">", "\\>"),       // Range: Indicates greater-than range.
    ("=", "\\="),       // Range: Indicates equality.
    ("+", "\\+"),       // Sign: Indicates that the term must appear in the documents.
    ("-", "\\-"),       // Sign: Indicates that the term must not appear in the documents.
    ("&&", "\\&&"),     // Boolean: Logical AND operator.
    ("||", "\\||"),     // Boolean: Logical OR operator.
    ("!", "\\!"),       // Boolean: Logical NOT operator.
    ("~", "\\~"),       // Proximity: Indicates proximity searches.
    ("?", "\\?"),       // Wildcard: Represents a single character.
    ("\\", "\\\\"),     // Escape: Used to escape special characters.
    ("/", "\\/"),       // Regex: Used to begin or end a regular expression.
]);

Very readable, and hard to make a mistake. (I wish we could destructure consts into two lists, but we cant.)

I was wondering if it would would interest you incorporate this directly into aho-corasick? And generally what your thoughts are on multiple lists, vs lists of tuples.

Thanks!

[BurntSushi]

Thanks for reaching out. I did find your framing here a little confusing because AhoCorasick::replace_all only accepts one slice. But you mention two slices. After I read through your full post, I think I understand: you're talking about the original sequence of patterns (used to build AhoCorasick) and then the sequence of replacements for each pattern. But these are two entirely separate parts of the API.

Combining them into one API seems like bad juju to me, because it means you'll have to pay for AhoCorasick construction every single time you call it. Building a AhoCorasick can be somewhat expensive, so I'm not sure there's really enough benefit here to be worth that cost.

Moreover, I think that if lining up the patterns and the replacements is proving tricky for you, and you don't mind paying the construction cost of AhoCorasick every time you do a replacement, then it's perfectly fine for you to have your own convenience routines to do it. But they strike me as inappropriate to encourage users of this library to do that.

Note also the existence of AhoCorasick::replace_all_with, whcih may or may not help your specific use case. (It absolves you of the need to have a slice of replacements, for example.)

[BurntSushi]

Also, on a more philosophical note, I'll say that I don't find this style of panic to be worth worrying too much about. Unless you're building your patterns and replacements dynamically in a way that doesn't guarantee they are of the same length by construction, you're going to get a panic every time you use this API when you call it. It's an immediately apparent bug with a loud failure mode. Those are good.

[arifd]

Ah yes, I'm sorry!! My comments about the API taking in a list of tuples is totally misleading and erroneous on my part! I'm sorry I didn't think longer before suggsting it!

For context, this is how I'm building up my AC:

static AC: OnceLock<AhoCorasick> = OnceLock::new();
AC.get_or_init(|| {
    AhoCorasick::builder()
        .kind(Some(AhoCorasickKind::DFA))
        .build(REPLACER.0)
        .unwrap()
})
.replace_all(haystack, &REPLACER.1)

Of course, a list of tuples could/should not be the API for this at all. The convenience function is useful though, and yes it's up to you if you want to lift it into aho-corasick I was motivated to ask this, because I am copy pasting it into every crate that uses aho-corasick.

[BurntSushi]

Which thing exactly are you copy & pasting? What is the specific thing you are proposing? Like, can you show the API you would want added?

[arifd]

This util fn:

const fn aho_corasick_builder<'a, const N: usize>(
    tuples: [(&'a str, &'a str); N],
) -> ([&'a str; N], [&'a str; N]) {
    let mut lhs = [""; N];
    let mut rhs = [""; N];
    let mut i = 0;

    while i < N {
        lhs[i] = tuples[i].0;
        rhs[i] = tuples[i].1;
        i += 1;
    }

    (lhs, rhs)
}

I need in two places. I much prefer to define my search/replace patterns as lists of tuples.

But yes, the more I think about it, the less I think aho-corasick cares for my very speecific yet generic problem. It's just that, in my workspace now, two crates are using aho-corasick for string replace, and yes I absolutely could move this fn into a crate I own called util and to be fair this is problem is not unique to aho-corasick, so why should aho-corasick own this fn?

But my second (current) usecase for aho-corasick is in a proc macro crate, and I feel a little bit weird about a highly specific proc-macro crate, having dependencies scattered across my workspace.

Thirdly, i just feel a little bit embarrassed about wasting your time. I may have pulled the trigger a little bit too early in opening up this discussion, and there's no way to turn back time, unfortunately.

[BurntSushi]

Yeah that function has no place in aho-corasick specifically. But no worries. I appreciate folks asking questions. It's cool. I'm happy to spend time answering them.

FWIW, something that might put you at ease is that I regularly copy tiny functions like that across crates without evening batting an eye. DRY is overrated. :-)