[ENH]improve conditional join performance

[samukweku]

PR Description

Please describe the changes proposed in the pull request:

• improve performance when using numba and join conditions are more than two

This PR relates to #1415 .

In [2]: import pandas as pd; import janitor as jn; import numpy as np; import numba

In [3]: def east_west():
   ...:     num_rows_left, num_rows_right = 5_000_000, 5_000_000
   ...:     rng = np.random.default_rng(42)
   ...:
   ...:     # Generate two separate datasets where revenue/cost are linearly related to
   ...:     # duration/time, but add some noise to the west table so that there are some
   ...:     # rows where the cost for the same or greater time will be less than the east table.
   ...:     east_dur = rng.integers(1_000, 10_000_000, num_rows_left)
   ...:     east_rev = (east_dur * 0.123).astype(np.int32)
   ...:     west_time = rng.integers(1_000, 500_000, num_rows_right)
   ...:     west_cost = west_time * 0.123
   ...:     west_cost += rng.normal(0.0, 1.0, num_rows_right)
   ...:     west_cost = west_cost.astype(np.int32)
   ...:
   ...:     east = pd.DataFrame(
   ...:         {
   ...:             "id": np.arange(0, num_rows_left),
   ...:             "dur": east_dur,
   ...:             "rev": east_rev,
   ...:             "cores": rng.integers(1, 10, num_rows_left),
   ...:         }
   ...:     )
   ...:     west = pd.DataFrame(
   ...:         {
   ...:             "t_id": np.arange(0, num_rows_right),
   ...:             "time": west_time,
   ...:             "cost": west_cost,
   ...:             "cores": rng.integers(1, 10, num_rows_right),
   ...:         }
   ...:     )
   ...:     return east, west
   ...: east,west = east_west()


# dev
%timeit eastt.conditional_join(westt, ('dur','time','<='),('rev','cost','>='), ('id','t_id', '>'), use_numba=True)
11.3 s ± 55.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)


# this PR
%timeit east.conditional_join(west, ('dur','time','<='),('rev','cost','>='), ('id','t_id', '>'), use_numba=True)
3.41 s ± 2.68 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Please tag maintainers to review.

• @ericmjl

[ericmjl]

🚀 Deployed on https://deploy-preview-1419--pyjanitor.netlify.app

[ericmjl]

@samukweku, I am not 100% sure of the changes here. Is there a performance improvement? If so, is there a performance comparison that you've done compared to the previous iteration?

Apologies for getting back late here on my side!

[samukweku]

@ericmjl i have updated the example to show the perf. improvement

[ericmjl]

Ah, got it! Thanks @samukweku, I trust what you've got here. Let's merge! Please also do the honors of cutting a new release! I think we should just cut releases after merging PRs, so the improvements get shipped.

[samukweku]

@ericmjl it still needs an approval 😎

[codecov]

Codecov Report

Attention: Patch coverage is 88.94231% with 46 lines in your changes missing coverage. Please review.

Project coverage is 84.67%. Comparing base (6e77fbc) to head (fad36a8).
Report is 32 commits behind head on dev.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #1419      +/-   ##
==========================================
- Coverage   89.07%   84.67%   -4.41%     
==========================================
  Files          87       87              
  Lines        5374     6348     +974     
==========================================
+ Hits         4787     5375     +588     
- Misses        587      973     +386     

[ericmjl]

Ack, apologies @samukweku! Approved!