Update RAFT CI with pytorch 2.4.1 (facebookresearch#3980)

Summary:
Related to testing in facebookresearch#3974

Based on comparing the logs of two runs:
- failing: https://github.com/facebookresearch/faiss/actions/runs/11409771344/job/31751246207
- passing: https://github.com/facebookresearch/faiss/actions/runs/11368781432/job/31625550227

Pull Request resolved: facebookresearch#3980

Reviewed By: junjieqi

Differential Revision: D64778154

Pulled By: asadoughi

fbshipit-source-id: f4e53fed3850f3e0f391015c0349ee14da68330a

.github/actions/build_cmake/action.yml

@@ -63,7 +63,7 @@ runs:
           : # skip torch install via conda, we need to install via pip to get
             #  ROCm-enabled version until it's supported in conda by PyTorch
         elif [ "${{ inputs.gpu }}" = "ON" ]; then
-          conda install -y -q pytorch pytorch-cuda=12.4 -c pytorch -c nvidia/label/cuda-12.4.0
+          conda install -y -q "pytorch<2.5" pytorch-cuda=12.4 -c pytorch -c nvidia/label/cuda-12.4.0
         else
           conda install -y -q pytorch -c pytorch
         fi