JSON (de)serializer to fall back to something sensible (but not pickle)

[jtc42]

If a piece of data is not a primitive type, and cannot be serialized/deserialized using our sensible scientific defaults, it should fall back to a Pickle dumps with a bit of metadata about the system.

[ChasNelson1990]

I personally hate pickles. For many reasons. One of which is that there have been a variety of historical breaks of pickling files where they've change methods/defaults and future versions have got confused by old files and vice versa. Worth bearing in mind.

[jtc42]

If there's something functionally similar to pickle in terms of serialising "arbitrary" Python objects, I'm more than happy to use something else.

I guess the key point here is to have a half-sensible fallback serialisation method, and pickle is in the standard library. No particular love for it though!

[ChasNelson1990]

Okay, so using you against yourself, in #13 you talk about "it can be used outside of Python if you want", don't pickles go against that ideology?

See also: https://www.benfrederickson.com/dont-pickle-your-data/

Does the JSON module not have some sort of fallback built in? OR perhaps such types should'n't be serialised and should be blobs linked to from the JSON?

[jtc42]

Yeah absolutely. A truly platform agnostic way to serialise an arbitrary object would be ideal. However, realistically most data will eventually be stored in some kind of Numpy or Pandas array format, or a primitive type, which serialises nicely in a cross-platform way. This fallback would be for extreme fringe cases where it cannot be encoded in a platform agnostic format.

Even if we link from JSON to a binary file, we still need to find a way to encode the object into a standalone file. That's where I'm suggesting pickles, as they are designed for dumping arbitrary Python objects to a file.

If something more agnostic than pickle exists though, I'd love to use that. The aim is to support common data formats nicely, falling back to primitive data types, falling back in extreme cases to something like this.

[jtc42]

I think I'm going to close this as out-of-scope. We can realistically leave it up to developers to ensure their returned data types can be serialized.