Data import/export functionality

[smartalecH]

It would be great to have robust and standardized tooling for data import/export.

For example, it would be useful to be able to cache to disk LayerSolveResults scattering matrices, and amplitudes corresponding to a particular basis.

Pickle files aren't very robust (or secure). Perhaps Hdf5?

[mfschubert]

I have been using json for this:

https://github.com/invrs-io/totypes/blob/main/src/totypes/json_utils.py

[smartalecH]

Oh interesting... how big do these get? json isn't compressed or serialized in anyway.

[mfschubert]

They are about 1.4x the size of the object in memory. (I actually test for it: https://github.com/invrs-io/totypes/blob/main/tests/test_json_utils.py#L93)

Not a huge overhead, and the benefit is the file can be almost visually inspected.

[smartalecH]

Very cool! I'm moving this to a discussion, since there are probably a few different ways to tackle this, and this seems like a great option. Thank you!

[smartalecH]

@mfschubert suppose I want to cache a cascaded scattering matrix to disk (e.g. the result of cascading several discretized slabs of a microlens).

Naively, I might just dump the ScatteringMatrix data structure to disk using some aforementioned method. But this can require a LOT of storage if the LayerSolveResult contains a LOT of eigenvector data (e.g. imagine many fourier terms, over many wavelengths, over many k-points).

So maybe I get creative, and I just dump the scattering parameters themselves... after all, I can manipulate those with amplitudes rather easily.

But things get tricky if I want to dynamically cascade that scattering matrix with a new scattering matrix I compute on the fly. Because all the relevant LayerSolveResult data is gone (which is needed by _extend_s_matrix()).

In theory, we should be able to just Redheffer Star my cached S-matrix with my new S-matrix, and ignore my LayeredSolveResults. Meaning I can just add a function that performs this simple operation. Or am I missing something? This comment in the code made me pause:

fmmax/src/fmmax/scattering.py

Line 281 in 8c7491a

# Compute the interface matrices following equation 5.3 of [1999 Whittaker].

Is there more nuanced involved (e.g. matching boundary conditions)?

i.e. here's the logic I'm thinking:

1. Load in cached S-matrix (smatrix)
2. Compute the layered solve result for a new layer (solve_result_new)
3. Compute a new scattering matrix (smatrix_new)
4. Cascade the two using Redheffer Star (redheffer_star(smatrix,smatrix_new))

[mfschubert]

@smartalecH it would be nice to have this capability. Unfortunately, I believe that this may require the alternate form of the scattering matrix as discussed in the first reference of #71. I don't believe the scattering matrix of Whittaker appropriate for use with the star product. Caveat -- I'm not 100% sure about this.