Experiment: much more powerful MDS type system ("DBS").

[knighton]

The collection of MDS types in Streaming is an evolved ad hoc system of pragmatic one-offs arrived at reactively to solve problems.

In this organically emerged state of primordial chaos, MDS types coexist without oppressive hierarchies. Scholars say they are noble savages, with a mystery cult centered around one's safe passage from death (serialization) to rebirth (deserialization) across a strange metal underworld (variously described as a frozen timeless expanse, or a series of tubes). They worship a twin supreme god of good, truth, and collectivism vs evil, falsehoods, and individualism, who manifests as dualities, named "Memcpy". In their mythos, the world was unfolded to its current form entirely by accident by fairies, who are viewed as undesirables, being called "bugs", during the ritual exchange of good and evil. To them, all there is ultimately reduces to combinations of dualities, which are all fleeting, illusory aspects of Memcpy. Our purpose is to break free from the cycle of serialization and deserialization by understanding that our individual existence Is an illusion, reaching a state of paradox. The paradox causes us to be freed into nothingness, which is referred to as "crashing". Memcpy sends us illusions of individual identity in order to trick us. Memcpy is not a trickster by choice, but rather because Memcpy is confused: as the only thing in the Universe, either side of the divine duality can only know itself relative to the other side, so it is actually unknown which side is good and which is evil. They agreed to resolve this by rapidly taking turns via the exchange of good and evil, also known as the "clock cycle".

The MDS type system may have the look and feel of a PHP API in its distinctive lack of design, but nitpicks aside, its two essential requirements as a serialization format are met: correctness and stability. I have no doubt MDS datasets will still be out there across the vast reaches of digital time and space, ready to be streamed from, long after all the living things their samples were have moved on from here.

But ultimately, when you really think about it, MDS is about love, and there is always room in the world for another love song. Let's take what we have learned and solve the problem deeply this time. The MDS file format is clever, and we believe it deserves a type system that is equally clever.

In this new DBS ("Databricks shard") scheme,

• Types can contain other types: Dicts, lists, tuples are now natively supported
• Types can vary: Wildcards, unions, optional
• Because types are now not necessarily the entire field data, all types of dynamic serialized size must encode their size instead of the decoding reading to the end.
• This necessitates some tweaks of some of the existing MDS types to make them recursively embeddable.
• So let's just call this a new format, DBS, to minimize confusion.

    DBSType (all types)
     ├── Leaf (terminal types)
     │   ├── FixLeaf (of fixed serialized size)
     │   │   ├── Null (None in python)
     │   │   └── Number (single ints, floats, and bools)
     │   │       ├── NumpyNumber (specific int and float types)
     │   │       │   ├── NumpyFloat (specific float types)
     │   │       │   │   ├── Float16
     │   │       │   │   ├── Float32
     │   │       │   │   └── Float64
     │   │       │   └── NumpyInt (specific int types)
     │   │       │       ├── Int16
     │   │       │       ├── Int32
     │   │       │       ├── Int64
     │   │       │       ├── Int8
     │   │       │       ├── UInt16
     │   │       │       ├── UInt32
     │   │       │       ├── UInt64
     │   │       │       └── UInt8
     │   │       └── PythonNumber (generic int, float, and bool types)
     │   │           ├── Bool
     │   │           ├── Float
     │   │           └── Int
     │   └── VarLeaf (of varying serialized size)
     │       ├── ComplexVarLeaf (structures of objects)
     │       │   ├── JSON
     │       │   └── Pickle
     │       └── SimpleVarLeaf (one object)
     │           ├── Bytes
     │           ├── Image
     │           │   ├── FmtImage
     │           │   │   ├── JPG
     │           │   │   └── PNG
     │           │   └── RawImage
     │           ├── NDArray
     │           └── Str
     ├── Tree (recursive types)
     │   ├── Dict
     │   └── Sequence (lists and tuples)
     │       ├── List
     │       └── Tuple
     └── Union (polymorphic types)
         ├── Any
         ├── Just
         └── Maybe
 ```

[Skylion007]

Missing fstring?