TODO: meta-issue

[tlambert03]

General Issue for tracking ideas. These can be pulled out into sub-issues for more focused discussion. They are in no particular order

• Improve backend abstractions. A goal is to be able to take the most advantage of any hardware. PyTorch does pretty well for apple-silicon via metal. Jax can too... but support for mps is a bit more experimental, and some things like FFTs aren't yet implemented. Everything works fine for nvidia of course, cupy provides very good performance and compatibility in that case. We don't necessarily need to support all 4 of those backends; the general goal is to have great performance on any device. By far the most important functionality will be FFTs and convolution, so we really don't need much from each backend. Opencl via pyopencl should be considered as well, and pyvkfft gives great FFT performance there.
• deal with very large ground truths and PSFs. Most modalities require 3D convolutions, and as the ground truth gets large, that can be very hard. We can experiment with using tiled convolutions; using dask.array.map_overlap may be useful here, but as the kernel size itself gets very large, we'll be unable to do FFT-based convolutions. whether we need large kernels is an open question: there may be some reasonable tradeoff between realism (large PSF size) and performance (small PSF size), particularly depending on the final SNR of the image.
• experiment with alternatives to FFT-convolutions. It would be nice to have a better sense for when real-space convolutions are preferably, or when we can get clever about image formation strategies
• more psf models. Currently we only implement gibson-lanni style PSFs. We should implement full zernike polynomial backed PSFs, various places to look include chromatix, pyotf, napari-psf-simulator
• more modalities: light sheet, 2P, localization (STORM/PALM), STED, etc...
• Integrate fluorophore into the simulation.  #22
• bleaching/temporal models
• motion models
• refactor: remove DataArray, use xarray directly, vendor xarray_jax from deepmind/graphcast #43 - xarray's DataArray. Currently, I have a DataArray object that implements the bare minimal xarray interface that i want to use. The reason we want something like this in the first place is that it makes for a nice object to pass through the simulation train, in that it contains axis dimension information and spacings/transforms (rather than passing a ton of stuff along with the array object). I didn't use xarray directly because I was a bit worried that it was casting gpu-arrays to numpy arrays under the hood. but I think I'm wrong about that. Later tests confirmed that it can do a pretty good job with many array-likes (meaning it uses their buffers under the hood and doesn't copy data). It does great with cupy, but not so immediately great with pytorch. Jax can kinda work, but since jax arrays are immutable, special considerations will need be made in our numpy-backend for in-place mutations.
• finish grabbing ground-truth data from cosem. The goal there would be for an end-user to just say "i want to see EGFP-labeled ER and mCherry-labeled mitochondria", and we would then grab a reasonable chunk of ground truth from open-organelle, and fluorophore properties from fpbase.
• build more simulated ground truth objects. This could be anything: ellipsoids that behave like nuclear envelope, curvilinear microtubule thingies. Lots of fun to be had making up ground truth models.
  • create Sample for cosem data from feat: add cosem models, downloading, caching, binning, and tests #50
  • create Sample for allen data from feat: pull neuron ground truths from allen brain #48
• more examples and "recipes" for useful way to use the library.
• Add more detectors: generic PMTs, APDs, HyDs... and provide a library of presets so that someone could do something like detector = 'ORCA-Quest'