Ensure explicit output dtype for pad_across_processes

[mariusarvinte]

What does this PR do?

Fixes #3218.

Current implementation casts torch.bool to torch.int64 because of + pad_index, where pad_index is 0 by default:

accelerate/src/accelerate/utils/operations.py

Line 672 in d0e80e5

new_tensor = tensor.new_zeros(tuple(new_size)) + pad_index

Adds a test case for checking that torch.bool is output with the same type.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@muellerzr @BenjaminBossan @SunMarc

[BenjaminBossan]

Hmm, I'm wondering how safe this is in case that the tensor dtype cannot represent the new data. E.g. when pad_index is not 0 or 1, casting this to bool will result in a loss of information.

[SunMarc]

+1, pad_index in the context of LLMs is usually -100.

[mariusarvinte]

I don't think there's any loss of useful information per se, given that the original data is retained downstream

accelerate/src/accelerate/utils/operations.py

Line 679 in c0552c9

new_tensor[indices] = tensor

What this does change though is the actual pad value. Any non-zero pad_index (e.g., -100) will result in padding with True for bool.

Does it make sense to always pad with False for bool? In our usecase, we directly manipulated bool tensors across devices and left pad_index = 0 by default. Not sure if bool actually appears in LLMs.