Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix VBE+SSD path with StagedPipeline #2607

Closed
wants to merge 1 commit into from

Conversation

chrisxcai
Copy link
Contributor

Summary:
when SSD offloading is configured and VBD is used, as in Jupiter XL case:

we will call into

if sparse_features.variable_stride_per_key() and len(embeddings) > 1:
            embeddings = self._merge_variable_batch_embeddings(embeddings, vbe_splits)

and throw the error:

File "/mnt/xarfuse/uid-179947/2baf53ce-seed-nspid4026531836_cgpid5102251-ns-4026531841/torchrec/distributed/embedding_lookup.py", line 530, in <listcomp>
    split_embs = [e.split(s) for e, s in zip(embeddings, splits)]
  File "/mnt/xarfuse/uid-179947/2baf53ce-seed-nspid4026531836_cgpid5102251-ns-4026531841/torch/_tensor.py", line 1028, in split
    return torch._VF.split_with_sizes(self, split_size, dim)
RuntimeError: split_with_sizes expects split_sizes to sum exactly to 16 (input tensor's size at dimension 0), but got split_sizes=[4096]

need to make sure we pass batch_size_per_feature_per_rank for SSDTableBatchedEmbeddingBags case as well, otherwise the embedding output shape won't be in 1-D as expected

Reviewed By: dstaay-fb, sryap

Differential Revision: D66647146

Summary:
when SSD offloading is configured and VBD is used, as in Jupiter XL case:

we will call into
```
if sparse_features.variable_stride_per_key() and len(embeddings) > 1:
            embeddings = self._merge_variable_batch_embeddings(embeddings, vbe_splits)
```

and throw the error:

```
File "/mnt/xarfuse/uid-179947/2baf53ce-seed-nspid4026531836_cgpid5102251-ns-4026531841/torchrec/distributed/embedding_lookup.py", line 530, in <listcomp>
    split_embs = [e.split(s) for e, s in zip(embeddings, splits)]
  File "/mnt/xarfuse/uid-179947/2baf53ce-seed-nspid4026531836_cgpid5102251-ns-4026531841/torch/_tensor.py", line 1028, in split
    return torch._VF.split_with_sizes(self, split_size, dim)
RuntimeError: split_with_sizes expects split_sizes to sum exactly to 16 (input tensor's size at dimension 0), but got split_sizes=[4096]
```


need to make sure we pass batch_size_per_feature_per_rank for SSDTableBatchedEmbeddingBags case as well, otherwise the embedding output shape won't be in 1-D as expected

Reviewed By: dstaay-fb, sryap

Differential Revision: D66647146
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 4, 2024
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D66647146

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants