(ebnf) add backtick (grave symbol) as string character

[vancluever]

This adds backticks as an allowed character for terminal strings.

This is necessary for describing backslashes properly as using single or
double quotes seems to cause parsing issues with highlight.js. It also
has precedent in some EBNF-described specifications such as the Go
specification.

[egor-rogov]

Indeed, although the wikipedia article on EBNF and some EBNF standards I've searched have no mention on backticks, Golang specification (https://golang.org/ref/spec) uses them.
PR looks good to me. @vancluever, please add a changelog entry.

[joshgoebel]

Ping.

[vancluever]

Hey @egor-rogov and @yyyc514, sorry for the delays on this one. Will add an entry shortly.

[vancluever]

Changelog has been updated now!

[joshgoebel]

This is necessary for describing backslashes properly as using single or
double quotes seems to cause parsing issues with highlight.js. It also
has precedent in some EBNF-described specifications such as the Go
specification.

I’m not sure I follow what you mean here?

[vancluever]

I’m not sure I follow what you mean here?

This can be seen in the developer preview. Try doing a grammar that uses "\" or '\' and it should break parsing, even though there's no apparent reason why it should.

[joshgoebel]

@vancluever Still not sure I follow. Can you make a reproducible example? You can fork my jsfiddle template:

https://jsfiddle.net/ajoshguy/a1kntyg4/

Example worth 1000 words. :)

[vancluever]

@yyyc514 https://jsfiddle.net/ksv9Lqcz/ should give a good idea of what's going on!

[joshgoebel]

Yes, indeed. EBNF is just reusing hljs.APOS_STRING_MODE, hljs.QUOTE_STRING_MODE which have \[char] as an escape sequence, so in your example the string isn't terminated since the quotes are escaped.

Does EBNF not have such escape sequences in strings? Or are they only valid in some types of strings but not others?

[vancluever]

Can't comment for sure, but I don't know if there's any accepted standard, to be honest. But I have never really been able to find any sort of escaping standard, and I don't really know if a language for a context-free grammar should even have them.

Our Sentinel specification follows the example of the Go spec in the use of backticks, however, so we will need them regardless of whether or not we fix the escaping.

[joshgoebel]

Escaping is mentioned here: https://www.w3.org/Notation.html and specifically:

If a double-quote character is used inside a string, it must appear as the sequence " (backslash followed by double quote).

That's regarding BNF, but still...

[joshgoebel]

From Google's docs:

escaped_char     = `\` ( "a" | "b" | "f" | "n" | "r" | "t" | "v" | `\` | "'" | `"` ) 

I think the backtick is specifically for avoiding the need to double escape \ as \\. They seem to only consistently use it to enclose the backslash and prefer "" and '' in other places.

Another example where it looks like it's used for escaping:

Engelberg/instaparse#114

So I'm thinking our existing behavior with normal strings is correct and NOT an issue.

@egor-rogov Any insight to add here?

[joshgoebel]

Holding this a bit longer since I'll adjust the commit text based on our conclusions here.

[egor-rogov]

I think the backtick is specifically for avoiding the need to double escape \ as \. They seem to only consistently use it to enclose the backslash and prefer "" and '' in other places.

I think so too.
But I see no harm in adding backticks as rightful string terminators. I think it's better than allow only backtick-backslash-backtick sequences.

[joshgoebel]

But I see no harm in adding backticks as rightful string terminators.

Nor do I, but the original commit message make it sound like Highlight.js is BROKEN with regard to our standard string handling here. So I'm just trying to clear up if we still have an actual problem or if that original message was in error so that I can write a proper one when I merge this. :)

[vancluever]

Thanks @yyyc514 and @egor-rogov!