Partial data

[rartino]

This pull request is work by me and @sauliusg to represents a rather long discussion at the meeting among @sauliusg, @JPBergsma , @giovannipizzi , @gmrigna , myself, and others.

It attempts to generate a flexible enough format with which it is possible to transmit list property data that is too big to fit in a single response.

We have decided to separate this from the related issues of providing functionality for a client to "slice" provided data. That will be done in a separate pull request, defining a property_ranges parameter to the "normal" OPTIMADE endpoints. The response to such a request may (but need not) lead to data being transmitted the way described here.

[merkys]

This proposal looks OK to me. I am only slightly worried about data values stepping on the reserved symbols/structures, i.e., values with \n seem to break JSON-lines format and values containing ["ref"] or ["end"] may conflict with the special markers. I can think of a way to avoid the latter conflict by replacing conflicting values by URL references. Another solution would be to allow the provider to specify strings replacing "ref" and "end" in the JSON-lines header (compare to boundary in multipart/form-data).

[rartino]

Attention stakeholders: @rartino @sauliusg @giovannipizzi @gmrigna @JPBergsma @merkys
The deadline previously set for this PR expires midnight TODAY, June 15, 2023.

This is how I intend to proceed:

• Ideally I'd like to see 5 approvals before midnight, at which point I will make a happy dance and merge the PR. If that is not the case, I will ignore approvals missing from stakeholders without clear open change requests, i.e., I will still merge, but with proportionally reduced levels of happiness.
• Should any new major change requests appear before the current deadline, I will move the deadline forward to 24h after my last merge of an (attempted) fix to provide everyone time to check if they agree with the change and re-approve. (However, minor editorial changes will be merged without a deadline extension).

I much appreciate your change requests on the format of explicit text edits rather than "How about we also...?"-type comments.

If you cannot adhere to the deadline as given, fixes are of course still possible after we've merged via additional PRs. Just make sure to get them in before we release v1.2.0.

One final thing: if you want a pretty-formatted version to read (of precisely the revision of the PR at the time I write this), you can find it via this link: https://github.com/Materials-Consortia/OPTIMADE/blob/4e9fb4d7423364ecb6b998e41fd9ec8a6efb0dff/optimade.rst#optimade-json-lines-partial-data-format

Edit: I'll keep a changelog here:

• Added a field to provide an inline schema (in addition to a link to it), which I had intended to add before making this post, but forgot (the pretty-formatted link above is updated).

[JPBergsma]

Should this not come from PR #463?

Suggested change

	- **meta**: a `JSON API meta object <https://jsonapi.org/format/1.0/#document-meta>`__ that is used to communicate metadata.
	See `JSON Response Schema: Common Fields`_ for more information about this field.

[rartino]

I think we need to define the meta field here to hold the "partial_data_links" key? Otherwise this PR would be inconsistent.

[JPBergsma]

I think I came across another commit which removed part of the definition of the metadata fields. So it looked like you forgot this piece, which is why I mentioned it.
Either both should be in or both should be out.

[rartino]

Earlier I indeed removed a segment here that defined the property_metadata subkey of meta, which I agree belong better in #463. But, the segment you have marked now defines the meta superkey we need for the partial_data_links subkey.

I'm confused over what you are asking for. Are you saying we absolutely should not mention meta with a link to 'JSON Response Schema: Common Fields' that defines meta -> partial_data_links; despite that with this PR that key is an absolutely vital part of the 'Entry Listing JSON Response Schema'?

[JPBergsma]

I have not slept properly in days and therefore have not made as much progress with reviewing as I wanted. I will continue tomorrow.

[giovannipizzi]

I left a comment with minor language edit, otherwise I approve!

[rartino]

@giovannipizzi , @merkys Thanks for getting your text fixes in before the deadline. Can you please do the Review->Accept so we can get this merged? Thanks!

[merkys]

I must admit I had not have enough time to look through the every detail of the proposal. However, I do believe the remaining bugs (if any) will become clear once people start implementing it. Thus in order to stop blocking the PR (and the release), I approve.

[rartino]

Thanks to everybody who contributed to this PR! Now lets turn our attention to #462 #463