Improve performance of VaryingCelestialTransform #954

Job	Run time
sdist_verify	20s
docs / Load tox environments	9s
allowed-fail-tests / Load tox environments	5s
tests / Load tox environments	5s
allowed-fail-tests / py311-devdeps (ubuntu-latest)	2m 12s
tests / py312 (ubuntu-latest)	4m 57s
tests / py311 (ubuntu-latest)	3m 53s
tests / py311-online (windows-latest)	5m 27s
tests / py310 (macos-latest)	9m 3s
tests / py310-oldestdeps (ubuntu-latest)	3m 18s
docs / build_docs-notebooks (ubuntu-latest)	3m 11s
publish	0s
notify	0s
	32m 40s

Provide feedback