Expanding macro arguments and interpolations, inside and outside quoted strings

[Rangi42]

In the examples below, let's say that X EQUS "a\"b" and \1 is c"d (as in mymacro c"d).

Behavior in 0.4.1 and previous:

• Both {X} and "{X}" would act like "a\"b": the contents of the interpolated symbol get concatenated directly, essentially string-escaping their contents.
• \1 would expand to c"d and "\1" would expand to "c"d": both syntax errors due to unbalanced quotes, since macro args would always be expanded and then lexed further, as if they had been literally typed where the \ expansion was.

In 0.4.2 unquoted {X} became a syntax error.

As of 2020-12-28, the post-0.4.2 master behavior is an attempt to have useful behavior as needed: when inside a quoted string (i.e. when readString is doing the lexing), interpolation and macro arg contents are concatenated as-is / i.e. string-escaped; outside a quoted string, they're not. So:

• {X} expands to the syntax error a"b, but "{X}" expands to "a\"b"
• \1 expands to the syntax error c"d, but "\1" expands to "c\"d"

However, this still has the potential for awkward behavior when you try to open or close a string with an interpolation or macro arg. For example, if Y EQUS "\"hello", then printt {Y} world" will print hello world; but if W EQUS "world\"", then printt "hello {W} will give an Unterminated string error before printing hello world", because the final quote in W is concatenated to the string contents, not used to close the string. Doing printt {Y} {W} gives the same result; and using macro args instead of interpolations has the same issue. Various constructions involving triple-quoted string literals also give errors like this, which are predictable given the current implementation but are probably not useful.

There's another inconsistency: macro arguments are lexed in raw mode, which doesn't handle the difference between quoted and unquoted expansions in the same way. For example:

equsprint EQUS "printt"
myprint: MACRO
	printt \1
ENDM
Q EQUS "hello\\n"
	printt "{Q}"
	myprint "{Q}"
	equsprint "{Q}"

This prints:

hello\nhello
hello\n

[Rangi42]

I propose something closer to the 0.4.1 behavior, where macro args and interpolation are both always expanded and then lexed further; but with a way to explicitly request that the contents be taken directly / string-escaped.

This is easy to do for interpolation (should be added post- #646 though): using a format specifier can escape the contents, otherwise not. So "{s:X}" would expand to "a\"b" (and "{5s:X}" would expand to " a\"b", and so on for the new fancy interpolation specs), but plain "{X}" would go back to 0.4.1-style expansion and act like "a"b". (Alternatively, have two specifiers, like {s:X} for escaped and {v:X} (or {S:X} or {e:X} or whatever) for unescaped or vice-versa, with plain {X} defaulting to unescaped.)

For macro args, it would need something further. C has its # syntax to stringify macro arguments (as well as ## to concatenate them but STRCAT should cover those cases). There are probably a lot of potential syntaxes here:

• \01, \~1, or some other leading character (but this would have to coexist with line continuations and comments, like \0 ; end of line (newline) 1 ; second line)
• STRARG(1), which has the advantage of taking any numeric value (and you can still do raw "\1" if you assume the arg will contain all sane characters)
• Something else?

[Rangi42]

My own preference would be for plain {X} and \1 to go back to their 0.4.1 behavior, unescaped in all contexts. Then make {s:X} expand to the string-escaped contents of X, and add STRARG(1) to return the string contents of \1.

(So for a guaranteed valid string, you could do "hi {s:X} bye", or STRCAT("hi ", STRARG(1), " bye").)

This would avoid breaking asm code, since both quoted and unquoted macro args and interpolations would work the same as in 0.4.1 and 0.4.2. (Except unquoted interpolations, which acted like quoted ones in 0.4.1, and were banned in 0.4.2, and would now expand just like macro args.)

It would also allow the lexer to be simplified to not make readString special-case its interpolation and macro arg handling.

[Rangi42]

Another rationale for not introducing an explicit format spec type for escaped strings:

STRFMT("%d", N) is just like "{d:N}". STRFMT("%.5f:_PI") is just like "{.5f:_PI}". In general STRFMT("%<spec>", X) would make sense substituted for "{<spec>:X}". If X EQUS "a\"b", then STRFMT("%s", X) is the same string as "a\"b", so it should act just like "{s:X}".

Meanwhile, the plain untyped "{X}" has no possible equivalent STRFMT placeholder, since you need a letter for those; but one wouldn't make sense anyway. STRFMT("%?", X) should never turn into the syntax error "a"b" for some type ?. So, using the untyped notation {X} to get the unescaped string value makes sense, and avoid excessively changing behavior since 0.4.1 and 0.4.2.

[Rangi42]

Regarding STRARG: IMO this also makes sense as opposed to a lexer-time solution like \~1. Compare: _NARG is lexed as an ordinary identifier, which the parser then evaluates to the number of macro args if it's within a macro. Likewise, STRARG(n) would be an ordinary string function like STRUPR or STRFMT, which happens to return a result only if it's within a macro.

[Rangi42]

On the other hand, {s:X} for the string escaped-contents of X doesn't really make sense if it's not inside a string.

[Rangi42]

While I'm throwing out alternatives for discussion: consider if {X} and \1 both went back to their 0.4.1/0.4.2 behavior, which is the same whether or not it's inside readString, but add two new functions:

• STRARG(n): returns the contents of \n as a string (n is a uconst)
• STRSYM(sym): returns the contents of sym as a string (sym is a scoped_id, so like PURGE and EXPORT, it won't get expanded if it's an EQUS)

Thus if X EQUS "\"h", then PRINTT {X}i" expands to PRINTT "hi"; PRINTT "{X}i" expands to the syntax error PRINTT ""hi"; but PRINTT STRCAT(STRSYM(X), "i") will print "hi.

(So if you're confident that a macro arg or an EQUS symbol won't have any special characters in it, then you can just do "\1" or "{X}" and it will work as expected. Which as of 0.4.2 is your only choice.)

[ISSOtm]

I agree that mac "Hello, World!" should be a single argument, as I've seen a lot of people fall into the trap, both newbies and some regulars. The finding of #683 strengthens this need.

However, I've pondered a lot on how to transition into that change, and I haven't found a satisfactory answer yet. This is not behavior that I want to directly change (so I'm embarrassed that @Rangi42 opened #685 so soon), but I'm not sure how to deprecate it either. Release 0.4.3(-pre?) with a known bug and deprecation warning, and fix it properly in 0.4.4?

[Rangi42]

master has already added {interpolation} outside strings, and special handling of macro args and interpolations inside string literals to directly paste their contents. I think those two features would be (a) useful enough in metaprogramming and (b) incompatible enough with current macro arg handling, that it's worth fixing this in the next release, even if that means calling it 0.4.3-pre. (Re: deprecation rather than a breaking comment, I commented below on that.)

[Rangi42]

Is #685 unsatisfactory besides being a breaking change? It could be made more transition-friendly by having string literals allow \,, printing a deprecation warning instead of "Illegal character escape"; that would take care of users currently doing mymacro "Hello\, world!". Any users who currently pass incomplete string literals would be forced to edit their code, but I really doubt that's done in practice: even in rgbds's test cases the only ones covering that were recently introduced to help test """multiline strings""".

[Rangi42]

The next release will already change how string literals get handled, since if \1 is a", "b, then doing "p\1q" used to lex like "pa", "bq" but will now lex like "pa\", \"bq". May as well put the breaking changes together into one release? (I'd still be OK with/in favor of a deprecation warning for \, in strings, which is probably the most common changed behavior that's not just a cooked-up test case.)

[Rangi42]

A summary of my intended implemented behavior: #685 (comment)