Partial data

.words.lst

@@ -1,4 +1,4 @@
-personal_ws-1.1 en 205
+personal_ws-1.1 en 209
 ABNF
 ACM
 Aa
@@ -86,6 +86,7 @@ bandgap
 bd
 booktitle
 boolean
+bzip
 calc
 cartesian
 checksums
@@ -115,18 +116,21 @@ exclusiveMinimum
 exmpl
 fieldname
 firstname
+hdf
 howpublished
 href
 html
 http
 hydrogens
 hydroperoxide
 implementers
+incrementing
 internaldb
 javascript
 json
 jsonapi
 jsonc
+jsonlines
 kvak
 lastname
 libc
@@ -203,4 +207,4 @@ xy
 yacc
 zeo
 zeolites
-�ngstr�m
+ångström

optimade.rst

@@ -593,6 +593,23 @@ Every response SHOULD contain the following fields, and MUST contain at least :f
 - **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON API resource object <http://jsonapi.org/format/1.0/#document-resource-objects>`__ or a *list* of JSON API resource objects.
   Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field.
 
+  The :field:`data` field MAY also contain a :field:`meta` field with the following keys:
+
+  - **property_metadata**: an object containing per-entry and per-property metadata.
+    The keys are the names of the fields in :field:`attributes` for which metadata is available.
+    The values belonging to these keys are dictionaries containing the relevant metadata fields.
+
+  - **partial_data_urls**: an object used to list URL:s which can be used to fetch data that has been omitted from the :field:`data` part of the response.
+    The keys are the names of the fields in :field:`attributes` for which partial data URLs are available.
+    Each value is a list of items that MUST have the following keys:
+
+    - **format**: String.
+      A name of the format provided via this URL.
+      One of the items SHOULD be "json lines", which refers to the format in `OPTIMADE JSON lines partial data format`_.
+
+    - **url**: String.
+      The URL from which the data can be fetched.
+
 The response MAY also return resources related to the primary data in the field:
 
 - **links**: `JSON API links <http://jsonapi.org/format/1.0/#document-links>`__ is REQUIRED for implementing pagination.
@@ -915,7 +932,8 @@ OPTIONALLY it can also contain the following fields:
 
   - **self**: the entry's URL
 
-- **meta**: a `JSON API meta object <https://jsonapi.org/format/1.0/#document-meta>`__ that contains non-standard meta-information about the object.
+- **meta**: a `JSON API meta object <https://jsonapi.org/format/1.0/#document-meta>`__ that is used to communicate metadata.
+  See `JSON Response Schema: Common Fields`_ for more information about this field.
 
 - **relationships**: a dictionary containing references to other entries according to the description in section `Relationships`_ encoded as `JSON API Relationships <https://jsonapi.org/format/1.0/#document-resource-object-relationships>`__.
   The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON API resource identifier object.
@@ -3200,6 +3218,163 @@ Relationships with files may be used to relate an entry with any number of :entr
 Appendices
 ==========
 
+OPTIMADE JSON lines partial data format
+---------------------------------------
+The OPTIMADE JSON lines partial data format is a lightweight format for transmitting property data that are too large to fit in a single OPTIMADE response.
+The format is based on `JSON Lines <https://jsonlines.org/>`__, which allows for streaming handling of large datasets.
+
+To communicate a property using this format, the usual OPTIMADE response gives the value :val:`null` for the property.
+Furthermore, a URL is given which can be used to fetch the missing data.
+For responses that use the JSON response format, a subfield :field:`partial_data_urls` of the resource object metadata field, :field:`meta`, is used, see `JSON Response Schema: Common Fields`_.
+
+.. _slice object:
+
+To aid the definition of the "json lines" format below, we first define a "slice object" to be a JSON object describing slices of arrays.
+The dictionary has the following OPTIONAL fields:
+
+- :field:`"start"`: Integer.
+  The slice starts at the value with the given index (inclusive).
+  The default is 0, i.e., the value at the start of the array.
+- :field:`"stop"`
+  The slice ends at the value with the given index (inclusive).
+  If omitted, the end of the slice is not specified.
+  If the end of the slice is not specified when used to express the values included in a response, the client has to count the number of items to know the end.
+  If the slice refers to a requested range of items, to omit :field:`stop` has the same meaning as specifying the last index of the array.
+- :field:`"step"`
+  The absolute difference in index between two subsequent values that are included in the slice.
+  The default is 1, i.e., every value in the range indicated by :field:`start` and :field:`stop` is included in the slice.
+  For example, a value of 2 denotes a slice of every second value in the array.
+
+Furthermore, we also define the following special markers:
+
+- The "end-of-data--marker" is this exact JSON: :val:`[["end"], ""]`.
+- A "reference-marker" is this exact JSON: :val:`[["ref"], "URL"]`, where :val:`"URL"` is to be replaced with a URL being referenced.
+- A "next-marker" is this exact JSON: :val:`[["next"], "URL"]`, where :val:`"URL"` is to be replaced with the target URL for the next link.
+
+These JSON markers have been deliberately designed as lists with items of mixed data types, and thus cannot be encountered inside the actual data of an OPTIMADE property.
+
+The full response MUST be valid `json lines <https://jsonlines.org/>`__ that adheres to the format:
+
+- The first line is a header object (defined below)
+- The following lines are data lines adhering to the formats described below.
+- The final line is either an end-of-data--marker (indicating that there is no more data to be given), or a next-marker indicating that more data is available, which can be obtained by retrieving data from the provided URL.
+
+The first line MUST be a JSON object providing header information.
+The header object MUST contain the key:
+
+- :field:`"format"`: String.
+  A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the returned format is dense or sparse.
+
+The header object MAY also contain the key:
+
+- :field:`"returned_ranges"`: Array of Object.
+  For dense data and sparse data of one dimensional list properties, the array contains a single element which is a `slice object`_ representing the range of data present in the response.
+  Once the client has encountered an end-of-data--marker, any data not covered by any of the encountered slices are to be assigned the value :val:`null`.
+  If the field :field:`"format"` is `"dense"` and :field:`"returned_ranges"` is omitted, then the client MUST assume that the data is a continuous range of data from the start of the array up to the number of elements given until reaching the end-of-data--marker or next-marker.
+  In the specific case of a hierarchy of list properties represented as a sparse multi-dimensional array, if the field :field:`"returned_ranges"` is given, it MUST contain one slice object per dimension of the multi-dimensional array, representing slices for each dimension that cover the data given in the response.
+
+The format of data lines of the response (i.e., all lines except the first and the last) depends on whether the header object specifies the format as :val:`"dense"` or :val:`sparse`.
+
+- **Dense format:** In the dense partial data format, each data line reproduces one list item in the OPTIMADE list property being transmitted in JSON format.
+  If OPTIMADE list properties are embedded inside the item, they can either be included in full or replaced with a reference-marker.
+  If a list is replaced by a reference marker, the client MAY use the provided URL to obtain the list items, which is then also provided in the JSON lines partial data format.
+
+- **Sparse format for one-dimensional list:** When the response sparsely communicates items for a one-dimensional OPTIMADE list property, each data line contains a JSON array on the format:
+
+  - The first item is the index of the item provided.
+  - The second item is a JSON representation of the item, on the same format as the lines in the dense format.
+    In the same way as for the dense format, reference-markers are allowed for data that does not fit in the response.
+
+- **Sparse format for multi-dimensional lists:** Specifically for the case that the OPTIMADE property represents a series of directly hierarchically embedded lists, the server MAY represent them using a sparse multi-dimensional format.
+  In this case, each data line contains a JSON array in the format of:
+
+  - All items except the last item are coordinates providing indices in the embedded dimensions in the order of outermost to innermost.
+  - The last item is a JSON representation of the item at those coordinates, on the same format as the lines in the dense format.
+    In the same way as for the dense format, reference-markers are allowed for data that does not fit in the response.
+
+
+Examples
+--------
+
+An example of an OPTIMADE JSON-API response that contains a link to a partial data protocol URL:
+
+.. code:: json
+   {
+       "data": {
+          "type": "structures",
+          "id": "2345678",
+          "attributes": {
+              "a": null
+          }
+       }
+       "meta": {
+           "partial_data_urls": {
+               "a": [
+                   {
+                       "format": "plain-jsonlines",
+                       "url": "https://example.db.org/assets/partial_values/structures/2345678/a/default_format"
+                   },
+                   {
+                       "format": "bzip2-jsonlines",
+                       "url": "https://example.db.org/assets/partial_values/structures/2345678/a/bzip2_format"
+                   },
+                   {
+                       "format": "hdf5",
+                       "url": "https://example.db.org/assets/partial_values/structures/2345678/a/hdf5"
+                   }
+               ]
+           }
+           "property_metadata": {
+               "a": {
+
+               }
+           }
+       }
+   }
+
+An example of a dense response for a partial array data, scalar values:
+
+.. code:: json
+    {"format": "dense", "returned_ranges": [{"start": 10, "stop": 20, "step": 2}]}
+    123
+    345
+    -12.6
+    [["next"], "https://example.db.org/value4"]
+
+An example of a dense response for a partial array data, multidimensional array values:
+
+.. code:: json
+    {"format": "dense", "returned_ranges": [{"start": 10, "stop": 20, "step": 2}]}
+    [[10,20,21], [30,40,50]]
+    [["ref"], "https://example.db.org/value2"]
+    [[11, 110], [["ref"], "https://example.db.org/value3"], [550, 333]]
+    [["next"], "https://example.db.org/value4"]
+
+An example of a sparse response for a partial array data with aggregated dimensions, single dimension array:
+
+.. code:: json
+    {"format": "sparse"}
+    [3,5,19,  [10,20,21,30]]
+    [30,15,9, [["ref"], "https://example.db.org/value1"]]
+    [["next"], "https://example.db.org/"]
+
+An example of a sparse response for a partial array data with aggregated dimensions, scalar values:
+
+.. code:: json
+    {"format": "sparse"}
+    [3,5,19,  10]
+    [30,15,9, 31]
+    [["next"], "https://example.db.org/"]
+
+An example of a sparse response for a partial array data with aggregated dimensions, multidimensional array:
+
+.. code:: json
+    {"format": "sparse"}
+    [3,5,19, [ [10,20,21], [30,40,50] ]
+    [3,7,19, [["ref"], "https://example.db.org/value2"]]
+    [4,5,19, [ [11, 110], [["ref"], "https://example.db.org/value3"], [550, 333]]
+    [["end"], ""]
+
 The Filter Language EBNF Grammar
 --------------------------------