LALR grammars

[joc-proxem]

I have a grammar that is successfully used with a Lark parser in 'lalr' mode. In this configuration, parsing is very fast.
When translated as a GBNF grammar, xgr.fill_next_token_bitmask may take several seconds on some states.
My guess is that XGrammar does not implement LALR states merging (Lark parser is slow as well when not in 'lalr' mode). Is it something that is planned ?

[Ubospica]

Hi @joc-proxem , thanks for the question! XGrammar does not implement LALR for now. The benefit is it can parse non-lalr grammar, although it will make parsing some lalr grammar slow. We now have a plan to enhance the grammar parser. Please stay tuned!