EEP 78: Multi-valued comprehensions

[michalmuskala]

TL;DR: This EEP proposes enhancing the comprehension syntax to allow emitting multiple
elements in a single iteration of the comprehension loop - effectively enhancing
comprehensions to implement flatmap with a fixed number of elements, for example:

[X + 1, X + 2, ... || X <- Xs]

[elbrujohalcon]

Not as a criticism, just out of curiosity… What will these expressions evaluate to?

[ Z = binary_to_integer(X), Z || X <- [<<"1">>, <<"2">>, <<"3">>] ].
%% My guess: [1, 1, 2, 2, 3, 3].
[ hello, goodbye || true ].
%% My guess: [hello, goodbye].
#{hello => Hello = hello, Hello => goodbye} || true }.
%% My guess: #{hello => goodbye}.

Maybe you can even add some of them as tests in erlang/otp#9374

[michalmuskala]

The first expression will be an error:

test.erl:6:31: variable 'Z' is unbound
%    6|   [ Z = binary_to_integer(X), Z || X <- [<<"1">>, <<"2">>, <<"3">>] ].
%     |                               ^

Similar how it would be in a plain list [X = binary_to_integer(~"1"), X].

The second will indeed yield as you expected:

3> [ hello, goodbye || true ].
[hello,goodbye]

The third will again error with Hello variable undefined:

test.erl:6:29: variable 'Hello' is unbound
%    6|   #{hello => Hello = hello, Hello => goodbye || true }.
%     |                             ^

The same would happen if you tried this with a plain map #{hello => Hello = hello, Hello => goodbye} won't compile.

In other words, similar to all "containers" expressions are evaluated "in parallel" not "in sequence" as far as variable scoping is concerned.

[Maria-12648430]

Hm, there is an unconvenient limitation, namely that it is not possible to decide if something should be inserted once, twice, thrice etc, or not at all (yes, that can and still has to be done with filters). That is, with the proposed syntax, it is only possible to add elements a fixed number of times, which IMO puts a severe limitation on the ergonomics the extension would otherwise provide.

That said, I also find it somewhat confusing. It looks like a set of expressions where, everywhere else, the last would be the result, but here the result of each expression is inserted.

[elbrujohalcon]

That said, I also find it somewhat confusing. It looks like a set of expressions where, everywhere else, the last would be the result, but here the result of each expression is inserted.

I guess you should think of it as the list of expressions at the "head" of a cons operator…

1> [1, 2, 3 | lists:seq(4, 10)].
[1,2,3,4,5,6,7,8,9,10]

I presume that's what @michalmuskala was using as inspiration.

[Maria-12648430]

I guess you should think of it as the list of expressions at the "head" of a cons operator…

Ok, makes sense :)

[Maria-12648430]

This is true, but also the multi-value syntax should be applicable to it, if only for the sake of consistency and completeness.

I think a binary comprehension like...

<< <<(X + 1)>>, <<(X + 2)>> || <<X>> <= Bin >>.

... should be accepted and result in a concatenation of the generated binaries, that is, same as if written <<(X + 1), (X + 2)>>.

[michalmuskala]

Yes, that's a good point. TBH the syntax for binary comprehensions is quite noisy already and I didn't want to muddle it more

[kikofernandez]

Hi @michalmuskala , thanks for your contribution.
Did you create a topic in the ErlangForums? I did not see, but maybe I was not looking too much in depth

[michalmuskala]

@kikofernandez I've created one now: https://erlangforums.com/t/eep-78-multi-valued-comprehensions/4537

[TD5]

The proposal as stated by @michalmuskala would be incredibly useful for me, since I have a lot of hot-code which ends up with precisely the style/performance trap described with:

lists:append([[X + 1, X + 2] || X <- Xs]

or

[Tmp || X <- Xs, Tmp <- [X + 1, X + 2]]

Hm, there is an unconvenient limitation, namely that it is not possible to decide if something should be inserted once, twice, thrice etc, or not at all (yes, that can and still has to be done with filters). That is, with the proposed syntax, it is only possible to add elements a fixed number of times, which IMO puts a severe limitation on the ergonomics the extension would otherwise provide.

I am not sure about an arbitrary number of elements, but for a closed-set of elements to be prefixed, I wonder whether something akin to this is possible:

[
  case X of
    one -> X + 1;
    two -> X + 1, X + 2;
    three -> X + 1, X + 2, X + 3
  end
|| X <- List]

Of course, this precise syntax is not a good idea at all, since there's now a strange new kind of expression in the tail position of a case-expression, but presumably we could devise some syntax for inside comprehension to conditionally determine the elements to emit.

[kikofernandez]

@kikofernandez I've created one now: https://erlangforums.com/t/eep-78-multi-valued-comprehensions/4537

I will merge the proposal into the EEP repo Monday next week. Just to leave some time in case there is more feedback from the community towards it :)

Thanks for your contribution!

[Maria-12648430]

Of course, this precise syntax is not a good idea at all, since there's now a strange new kind of expression in the tail position of a case-expression, but presumably we could devise some syntax for inside comprehension to conditionally determine the elements to emit.

It's not a new kind of expression, this is valid today, with the last expression of a case branch being the one that gets inserted in the list. Which is definitely worse in more than one regard than strange new expressions.

That said, I am also not a huge fan of introducing brand new syntaxes that only can be used in this particular context of comprehensions. I would argue that with things getting as complex as this, it is out of the scope of what is at the end of the day a shorthand syntax, and explicit recursion or a fold should be used.