best practice for patterns / lexers

[clach04]

First time playing with LPeg / scintillua so I may not know how to ask the right questions.

I'm working on a todo.txt lexer, I have something that works https://github.com/clach04/scintillua/blob/todotxt/lexers/todotxt.lua but it's the first working version and there are a few things I'm not sure about.

1. What's the best practice for chaining patterns with "starts with"? Nest or concatenate?

   For example:

lexer.starts_line(P('(') * lexer.upper * P(') '))

versus:

lexer.starts_line(P('(')) * lexer.upper * P(') ')

Any performance impact?

2. I don't fully understand; #patt Matches patt but consumes no input. I want to match dates in ISO like format, my initial pattern was too greedy and matches anything that has is date-like (see "order number" example at end):

lex:add_rule('date', lex:tag(lexer.KEYWORD, lexer.digit*lexer.digit*lexer.digit*lexer.digit*P('-')*lexer.digit*lexer.digit*P('-')*lexer.digit*lexer.digit))  -- too aggressive

I thought a prefix of #lexer.space (or even lexer.space) would help filter, it does NOT ☹️ .

lex:add_rule('date', lex:tag(lexer.KEYWORD, #lexer.space*lexer.digit*lexer.digit*lexer.digit*lexer.digit*P('-')*lexer.digit*lexer.digit*P('-')*lexer.digit*lexer.digit))
--lex:add_rule('date', lex:tag(lexer.KEYWORD, lexer.space*lexer.digit*lexer.digit*lexer.digit*lexer.digit*P('-')*lexer.digit*lexer.digit*P('-')*lexer.digit*lexer.digit))

However, post-fix works great 😃

lex:add_rule('date', lex:tag(lexer.KEYWORD, lexer.digit*lexer.digit*lexer.digit*lexer.digit*P('-')*lexer.digit*lexer.digit*P('-')*lexer.digit*lexer.digit*#lexer.space))  -- seems to work perfectly but I do not understand why prefix does not

3. I can't figure out how to create a non-whitespace word pattern. I was hoping I could use any word pattern and then negatively match white space, but it doesn't really work for me.

local not_whitespace = lexer.any - lexer.space

I'm currently using a workaround to match basic us-ascii (with some exceptions due to use case) but it feels dirty:

local not_whitespace = lexer.alnum + punct_not_colon

Any suggestions? I know I need to rename this 😜 if I keep this approach.

Here is my test todo.txt file:

(A) Call Mom @Phone +Family
(A) Schedule annual checkup +Health
(B) Outline chapter 5 +Novel @Computer
(C) Add cover sheets @Office +TPSReports
Plan backyard herb garden @Home
Pick up milk @GroceryStore
Research self-publishing services +Novel @Computer
Check status of order number 123456-12-11-121
Check status of order number 123456-12-11-33
write down friends birthday of 1901-02-03
buy new USB hardrive p:1
write up report p:5
p:3 something else to do
blocks:p1 put money in bank key:value
2025-01-12 note backup due:2025-01-31 hide:1 rec:1b rec2:+2w p:2 blocks:p5
x 2025-01-12 2025-01-12 calendar backup
x Download Todo.txt mobile app @Phone

Screenshot from bat

My current progress:

TL;DR I have something that works, what do I need to polish this and potentially contribute it to scintillua?

[orbitalquark]

1. What's the best practice for chaining patterns with "starts with"? Nest or concatenate?

Performance-wise it shouldn't matter. Nesting is more readable.

2. I don't fully understand; #patt Matches patt but consumes no input.
   [...]
   However, post-fix works great 😃

#patt means match patt, but allow the next matching thing to also start matching at the same position, rather than start matching after patt. For example, if you have the text "foobar", #P('foo') * lex:tag('word', lexer.alpha^1) will match and tag "foobar" as "word" instead of just "bar". I hope that makes sense.

Postfix #lexer.space means that you expect there to be a space after the pattern, but allow Scintillua to match that space with a different rule (i.e. the built-in whitespace matching rule for all lexers).

3. I can't figure out how to create a non-whitespace word pattern.

That ought to work. There are many lexers that use this construct: https://github.com/search?q=repo%3Aorbitalquark%2Fscintillua%20lexer.any%20-%20lexer.space&type=code

TL;DR I have something that works

Nicely done! That's the hardest part.

what do I need to polish this and potentially contribute it to scintillua?

I would take a look at other lexers, note the style and structure, and try to mimic them. If and when you're ready to contribute, you can mark your PR as ready for review, and we'll go through the review process.

[clach04]

Thanks for the quick response @orbitalquark !

I'm still cogitating over number 2....

But I finally figured out number 3, I needed to exclude colon for my use case! 🙃

local not_whitespace = lexer.any - lexer.space - P(':')

[clach04]

#136 ready for review. I know it's incomplete from a meta-data perspective, so I need some feedback.