QR factorization.

[Hobbes-Le-Chat]

I am trying to port a rnn code developed in torch. I am using crusher to make my main development and scaling before running in Frontier. I am having a hard time dealing with the math libraries in particular this one:

nn.init.orthogonal_(m[n], gain=gain * scale)
torch/nn/init.py", line 484, in orthogonal_
    q, r = torch.linalg.qr(flattened)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Calling torch.orgqr on a CUDA tensor requires compiling PyTorch with cuSOLVER. Please use PyTorch built with cuSOLVER support.

Do you have any suggestions about how to get around this?

[ausellis0]

Hi @Hobbes-Le-Chat,
What version of PyTorch do you have installed (pip3 show torch)? I believe that you may have a CUDA-based version of PyTorch. On Crusher, you can use this command (for nightly preview, https://pytorch.org/get-started/locally/):

module load cray-python/3.10.10
module load rocm/5.6.0
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.6 

Here is what I see with torch/2.2.0.dev20230922 + rocm/5.6:

>>> import torch
>>> A = torch.tensor([[.3,4.,5.], [-3., 4.7, .2], [0., 1., 10.]])
>>> A
tensor([[ 0.3000,  4.0000,  5.0000],
        [-3.0000,  4.7000,  0.2000],
        [ 0.0000,  1.0000, 10.0000]])
>>> Q, R = torch.linalg.qr(A)
>>> Q
tensor([[-0.0995, -0.9708, -0.2183],
        [ 0.9950, -0.0971, -0.0218],
        [-0.0000, -0.2194,  0.9756]])
>>> R
tensor([[-3.0150,  4.2787, -0.2985],
        [ 0.0000, -4.5588, -7.0670],
        [ 0.0000,  0.0000,  8.6608]])

Thanks!
Austin

[Hobbes-Le-Chat]

I installed my torch with this command:

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.2

This issue is old. I tried to fixed before with Magma but did not work so I waited to rocSolver and it looks It is still not working or I may be doing something wrong. Anyways I implemented my own QR but still I would like to know how to make it work in torch.

Here is the information about my installation.

Name: torch
Version: 2.0.1+rocm5.4.2
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: [email protected]
License: BSD-3
Location: /lib/python3.11/site-packages
Requires: filelock, jinja2, networkx, pytorch-triton-rocm, sympy, typing-extensions
Required-by: deepspeed, lightning, pytorch-lightning, pytorch-triton-rocm, torchaudio, torchmetrics, torchvision

It works when you do it in the cpu but I want the gpu version.

[jychang48]

@Hobbes-Le-Chat could you tell us how to reproduce your specific error? It looks like this is the old issue you were referring to:

https://discuss.pytorch.org/t/is-there-a-way-to-get-qr-supported-through-magma-for-torch-rocm5-4-2/180801

What example/test case was run? We'd like to try and reproduce this, and pass any useful information along to our PyTorch team

[jychang48]

Hi @Hobbes-Le-Chat

I ran this example in the RNN-Factory repo and reproduced your error:

python train.py --load_model "" --wandb "" --proj_dir "out" \
--data_file "" --data_type "dummy" --vocab_size 0 \
--ctx_len 128 --epoch_steps 1000 --epoch_count 20 --epoch_begin 0 --epoch_save 10 \
--micro_bsz 16 --n_layer 12 --n_embd 768 --pre_ffn 0 --head_qk 0 \
--lr_init 6e-4 --lr_final 1e-5 --warmup_steps 0 --beta1 0.9 --beta2 0.99 --adam_eps 1e-8 \
--accelerator gpu --devices 1 --precision bf16 --strategy ddp_find_unused_parameters_false --grad_cp 0

It looks like --accelerator gpu is causing the problem. It's possible that we're now running just QR on the host because I still see GPU activity in other parts of this run. Does this fix your problem?

[Hobbes-Le-Chat]

I am confused. Yes, QR with gpu does not work but it work if you do it in cpu

                    m[n] = m[n].cpu()  # Move tensor to CPU
                    nn.init.orthogonal_(m[n], gain=gain * scale)
                    m[n] = m[n].cuda()  # Move tensor back to GPU (if your setup supports it)     

Or one can do it in numpy. The problem was that I wanted to do the QR using GPU.

[jychang48]

@Hobbes-Le-Chat I looked into this a little more. It seems like lack of QR/hipSOLVER support on AMD GPUs was a known issue: pytorch/pytorch#99508 . I think the problem is you're still running on an outdated version of PyTorch that does not contain the specific PR fixing said issue: pytorch/pytorch#97370.

As @ausellis0 suggested earlier, you need to upgrade pytorch to the latest version:

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.6

Note that this is the preview/nightly version, and not the stable version. You might need to add --upgrade to indicate that you're upgrading the version of pytorch.

I tried this on my end, and successfully ran this command without having to remove --accelerator gpu from the command line:

python train.py --load_model "" --wandb "" --proj_dir "out" \
--data_file "" --data_type "dummy" --vocab_size 0 \
--ctx_len 128 --epoch_steps 1000 --epoch_count 20 --epoch_begin 0 --epoch_save 10 \
--micro_bsz 16 --n_layer 12 --n_embd 768 --pre_ffn 0 --head_qk 0 \
--lr_init 6e-4 --lr_final 1e-5 --warmup_steps 0 --beta1 0.9 --beta2 0.99 --adam_eps 1e-8 \
--accelerator gpu --devices 1 --precision bf16 --strategy ddp_find_unused_parameters_false --grad_cp 0

Please let us know if upgrading pytorch to the latest nightly build version works for you

[Hobbes-Le-Chat]

Hello,
I have a random question, by any chance do you know if triton is now working in AMD gpus?

[ckime-amd]

Yes, it is unrelated, but hopefully https://github.com/ROCmSoftwarePlatform
/ triton is what you're looking for?

[Hobbes-Le-Chat]

Yes, it is unrelated, but hopefully https://github.com/ROCmSoftwarePlatform / triton is what you're looking for?

Thanks, I'll test it right away.