Rep removal

[swith005]

Why are these changes needed?

This PR adds a transform for repetition removal in documents.

[touma-I]

partial review. Will spend more time tomorrow...

[touma-I]

Its it possible to add a diagram that shows how you are implementing a wrapper around the existing code ? Also, to confirm the cargo.lock, cargo.toml,rust/main.rs and rust/table.rs are "copy as is" from the original git repo. right ? What if that code changes and someone wants to upgrade the transform to work with newer code? Is that something we want to allow ?

[cmadam]

Can you please also add a test file for the pure python invocation of the program? (test_rep_removal_python.py). Its content would be almost identical to the test_rep_removal_ray.py, except that it would use the python transform launcher and config.

[swith005]

done

[shahrokhDaijavad]

Thanks, @swith005. Great job!

[shahrokhDaijavad]

Hi, @swith005 Looks like you have made a change in the directory for rust that results in an error when I run the notebook.
In the README, it says:

Compile the dedup_dataset binary from the dpk_rep_removal package dir:

cd dpk_rep_removal
cargo install 

it should be cd dpk_rep_removal/rust

Then, when I run the notebook, I get the error:
ERROR:root:Error occurred: /bin/sh: /Users/shahrokhdaijavad/Documents/data-prep-kit-outer/transforms/universal/rep_removal/dpk_rep_removal/target/release/dedup_dataset: cannot execute binary file

Because it is trying to find datasets in dpk_rep_removal/target/release/dedup_dataset, but the datasets are in:
dpk_rep_removal/rust/target/release/dedup_dataset

Please look into this. Thanks.

[touma-I]

I was unable to produce a proper wheel using line 160. I had to use the lines below. Can you please confirm that this also works for you:

[tool.setuptools.package-data]
"dpk_rep_removal.rust" = ["**"] 
"dpk_rep_removal.gpt2" = ["**"] 

[swith005]

@touma-I I don't understand what you mean "build a proper wheeling using line 160". i thought the command was make build-pkg-dist which should include the package data mentioned above for rep_removal.

can you elaborate how you're building the wheel?

[swith005]

OK tested this works :)

[touma-I]

what if the user is installing the package via pip install ? would they need to navigate to the folder in venv/lib/python3.11/site-packages/dpk_rep_removal/rust ?

[swith005]

@touma-I if installing the package via pip install, they would still need to build the pkg first with make build-pkg-dist correct? in that case, they should compile the dedup_dataset binary as instructed

[touma-I]

@swith005 No. We only run build-pkg-dist once before we push the image to pypi.Once the image is in pypi, users will do pip install data-prep-kit-transforms[all]. At this point, their packages are installed in their environment. So the cd dpk_rep_removal/rust will not work.

[swith005]

updated README to reflect install from git repo or pip install

[shahrokhDaijavad]

@swith005 Please see @touma-I 's comments above.

[touma-I]

@swith005 Per our discussion earlier, you need to create a new PR to further simplify how users interact with this transform. Ideally, you will run the cargo command within your own code during initiailization. The only requirements on the users would be to install rust. cc:@shahrokh