tests: prefer cache for missing config

[Borda]

What does this PR do?

moving import of potentially missing packages due to configuration conflicts inside the function computing reference matric to utilize the full power of cashing; see: #2335 (comment)

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

---

📚 Documentation preview 📚: https://torchmetrics--2414.org.readthedocs.build/en/2414/

[codecov]

Codecov Report

Merging #2414 (49e1faf) into master (c53ea94) will decrease coverage by 0%.
The diff coverage is n/a.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #2414   +/-   ##
======================================
- Coverage      69%     69%   -0%     
======================================
  Files         307     307           
  Lines       17352   17348    -4     
======================================
- Hits        11961   11957    -4     
  Misses       5391    5391