Punctuation in Synonyms

[nathanjfield]

I'm using Meilisearch in a live production environment for employee reports. As speed is essential when inputting reports, several people use shorthand when entering information. Examples include:

• QR = Quick Release
• M/C = Machine
• G/Box = Gear Box

And so on.

The problem is that when adding synonyms into Meilisearch, the punctuation, forward slashes in this case, are removed. So 'm/c' becomes 'm c'. Meaning that the synonyms for search don't work.

I'd like the indexing/synonyms to include punctuation marks.

[gmourier]

Hi @nathanjfield 👋

I think it has to do with the fact that the soft/hard separators of the tokenizer are not also considered in the synonyms. @ManyTheFish do you confirm?

Customizing the separators could be a solution so that Meilisearch can take / as a real character. We have a dedicated product discussion on that pain point.

Thanks!

[ManyTheFish]

Hello @gmourier and @nathanjfield,
Yes, the "/" is considered a separator so if your search for "G/Box" the real search will be "g box".
However, synonyms are normalized as the same, and so, "gear box" should be counted as a synonym of "g box". 🤔

[macraig]

Hello everyone 👋

We just released a 🧪 prototype that allows customizing tokenization and we'd love your feedback.

How to get the prototype?

Using docker, use the following command:

docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-tokenizer-customization-2

From source, compile Meilisearch on the prototype-tokenizer-customization-2 tag

How to use the prototype?

You can find all the details in the PR.

⚠️ We do NOT recommend using this prototype in production. This is for test purposes only.

Feedback and bug reporting when using this prototype are encouraged! Thanks in advance for your involvement. It means a lot to us ❤️

[macraig]

Hello everyone 👋

We have just released the first RC (release candidate) of Meilisearch containing this new feature!

You can test it by using:

• The release assets
• The Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.4.0-rc.0

You are welcome to leave your feedback in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on September 25th, 2023

⚠️ RC (release candidates) are not recommended for production

[macraig]

Hey folks 👋

v1.4.0 has been released! 🦓 You can now customize tokenization by adding or removing tokens from the separator tokens and non-separator tokens lists. ✨

Note:

📚 Separator tokens
📚 Non-separator tokens