flisp is not thread safe

[yuyichao]

I'm pretty sure people already know this but I'm just looking for new ways to crash julia with threading and I couldn't find this problem mentioned in any open issues (#10421 or #1790).

The following code easily causes a segfault in flisp

@threads for i in 1:10000
    parse("1")
end

Garding parse("1") with a lock makes the SegFault go away. However, I'm not sure adding a flisp lock is the right solution here. We call back to arbitrary julia code/runtime in macroexpand and doing that while holding a lock doesn't sound like a particularly good idea. (We also don't hold any lock when instantiating a stage function....)

What's the right solution here? Can we release the flisp lock when evaluating julia code? (can flisp handle that?) Or can flisp be made more thread safe?

@JeffBezanson

[JeffBezanson]

flisp is called in a totally stateless fashion, so I think the best approach is to bundle all of its runtime state into a context object and have each thread use a separate flisp context. However that will take a significant amount of work, plus use a fair amount of memory, so a lock is probably the right thing for now.

[yuyichao]

I realized that releasing the lock when evaluating julia code is not so different from a stack switch without copy_stack. However, it seems that flisp doesn't play well with task switch either. The following code segfault on current master without threading.

macro macro1()
    yield()
    nothing
end

@sync for i in 1:100
    @async macroexpand(:(@macro1))
end

If we can fix this, it should be straight forward to lock flisp and make it thread safe.

[JeffBezanson]

Yielding inside a macro expander should be an error.

[JeffBezanson]

We also disallow task switches inside finalizers, so this could be generalized to a single task switch critical region mechanism.

[yuyichao]

It is sometimes useful to print stuff in macros though. Disallowing task switch will make that impossible.

[JeffBezanson]

It would be nice to have an escape hatch, e.g. make printing to stderr synchronous.