Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Dagger's eager API #55

Open
jpsamaroo opened this issue Jul 4, 2021 · 1 comment
Open

Use Dagger's eager API #55

jpsamaroo opened this issue Jul 4, 2021 · 1 comment
Labels
enhancement New feature or request

Comments

@jpsamaroo
Copy link
Collaborator

After JuliaParallel/Dagger.jl#223 gets merged, Dagger's eager API (Dagger.@spawn) should be suitable for use in packages. I would recommend we use it for non-lazy computations in FileTrees so that we can get the benefits of Dagger 100% of the time.

@jpsamaroo jpsamaroo added the enhancement New feature or request label Jul 4, 2021
@DrChainsaw
Copy link
Collaborator

I was thinking on how to add this in the best way. I assume we want it to be opt-in to begin with?

Would it suffice with some convenience which basically does this:

julia> tt = maketree("root" => ["next" => [(name=string(x), value=1:10) for x in 'a':'k']]);

julia> function myvcat(x, y)
           sleep(1)
           vcat(x,y)
       end
myvcat (generic function with 1 method)

julia> @time fetch(reducevalues((v1,v2) -> Dagger.@spawn(myvcat(v1, v2)), tt));
  4.159940 seconds (110.59 k allocations: 5.883 MiB, 2.32% compilation time)

For example, letting exec also fetch EagerThunks and wrapping @spawn in some utility function, e.g. parallel(f) = (args...) -> Dagger.@spawn(f(args...))?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants