ArgumentError: array must be non-empty when deserializing a PythonCall object

[schlichtanders]

PythonCall's deserialization is somehow not enabled in Malt. This is surprising, as PythonCall implements its interface via MultipleDispatch on Serialization.deserialize

julia> using Malt, PythonCall
...

julia> worker = Malt.Worker()
...

julia> Malt.remote_eval_wait(worker, quote
           import Pkg; Pkg.add("PythonCall"); import PythonCall
       end)
...

julia> Malt.remote_eval_wait(worker, quote
           $(pydict())
       end)
ERROR: Remote exception from Malt.Worker on port 9280 with PID 251280:

ArgumentError: array must be non-empty
Stacktrace:
  [1] _throw_argerror(s::String)
    @ Base ./array.jl:383
  [2] pop!
    @ ./array.jl:1398 [inlined]
  [3] deserialize_cycle
    @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:818 [inlined]
  [4] deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, t::DataType)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:1497
  [5] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:878
  [6] deserialize(s::Serialization.Serializer{Sockets.TCPSocket})
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814
  [7] deserialize_expr(s::Serialization.Serializer{Sockets.TCPSocket}, len::Int64)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:1291
  [8] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:894
  [9] deserialize(s::Serialization.Serializer{Sockets.TCPSocket})
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814
 [10] (::Serialization.var"#5#6"{Serialization.Serializer{Sockets.TCPSocket}})(i::Int64)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:973
 [11] ntupleany(f::Serialization.var"#5#6"{Serialization.Serializer{Sockets.TCPSocket}}, n::Int64)
    @ Base ./ntuple.jl:43
 [12] deserialize_tuple(s::Serialization.Serializer{Sockets.TCPSocket}, len::Int64)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:973
 [13] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:857
 [14] deserialize(s::Serialization.Serializer{Sockets.TCPSocket})
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814
 [15] (::Serialization.var"#5#6"{Serialization.Serializer{Sockets.TCPSocket}})(i::Int64)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:973
 [16] ntupleany(f::Serialization.var"#5#6"{Serialization.Serializer{Sockets.TCPSocket}}, n::Int64)
    @ Base ./ntuple.jl:43
 [17] deserialize_tuple(s::Serialization.Serializer{Sockets.TCPSocket}, len::Int64)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:973
 [18] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:857
 [19] deserialize(s::Serialization.Serializer{Sockets.TCPSocket})
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814
 [20] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:920
 [21] deserialize
    @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:814 [inlined]
 [22] deserialize(s::Sockets.TCPSocket)
    @ Serialization ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Serialization/src/Serialization.jl:801
 [23] serve(server::Sockets.TCPServer)
    @ Main ~/.julia/packages/Malt/Z3YQq/src/worker.jl:78
 [24] main()
    @ Main ~/.julia/packages/Malt/Z3YQq/src/worker.jl:33
 [25] top-level scope
    @ ~/.julia/packages/Malt/Z3YQq/src/worker.jl:166
 [26] include(mod::Module, _path::String)
    @ Base ./Base.jl:495
 [27] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:318
 [28] _start()
    @ Base ./client.jl:552
Stacktrace:
 [1] unwrap_worker_result(worker::Malt.Worker, result::Malt.WorkerResult)
   @ Malt ~/.julia/packages/Malt/Z3YQq/src/Malt.jl:50
 [2] _wait_for_response(worker::Malt.Worker, msg_id::UInt64)
   @ Malt ~/.julia/packages/Malt/Z3YQq/src/Malt.jl:325
 [3] _send_receive
   @ ~/.julia/packages/Malt/Z3YQq/src/Malt.jl:336 [inlined]
 [4] #remote_call_wait#43
   @ ~/.julia/packages/Malt/Z3YQq/src/Malt.jl:422 [inlined]
 [5] remote_call_wait
   @ ~/.julia/packages/Malt/Z3YQq/src/Malt.jl:421 [inlined]
 [6] remote_eval_wait
   @ ~/.julia/packages/Malt/Z3YQq/src/Malt.jl:491 [inlined]
 [7] remote_eval_wait(w::Malt.Worker, expr::Expr)
   @ Malt ~/.julia/packages/Malt/Z3YQq/src/Malt.jl:492
 [8] top-level scope
   @ REPL[5]:1

Using Malt.DistributedStdlibWorker everything works.

[fonsp]

Maybe we need a Base.invokelatest? On both sides? (The serialize call in the worker and the deserialize call in the server) Can you try that?

If so, can you see if Distributed does the same? (Search for invokelatest in the Distributed source code)

[schlichtanders]

Nice catch!
indeed distributed seems to use invokelatest

• serialize https://github.com/JuliaLang/Distributed.jl/blob/af89e6cd76b2a613d061d55864b8fc7ec8215911/src/messages.jl#L181
• deserialize https://github.com/JuliaLang/Distributed.jl/blob/af89e6cd76b2a613d061d55864b8fc7ec8215911/src/process_messages.jl#L176

in Malt.jl the place seems to be these ones:

• serialization

  Malt.jl/src/shared.jl

  Line 41 in 24a764c

  serialize(io, msg_data)

• deserialization

  Malt.jl/src/Malt.jl

  Line 203 in 24a764c

  deserialize(io), true