Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add per entry and per property meta data field. #463

Merged
merged 43 commits into from
Jun 16, 2023
Merged
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
4170937
First draft metadata field.
JPBergsma Apr 6, 2023
92c4d92
updated example so the numbers make sense.
JPBergsma Apr 11, 2023
6fae33d
Added that metadata fields can have their own metadata.
JPBergsma Apr 11, 2023
d343a86
Add specifically that meta suffix must not be used other than for met…
JPBergsma Apr 18, 2023
54b632c
Moved database specific fields to a seperate section and renamed them…
JPBergsma Apr 18, 2023
5d186d9
Apply suggestions from code review
JPBergsma Apr 18, 2023
cb63bf2
Added 'metadata_for' field.
JPBergsma Apr 25, 2023
d0a4e65
Revert "Moved database specific fields to a seperate section and rena…
JPBergsma Apr 25, 2023
9e64a3a
Revert "Add specifically that meta suffix must not be used other than…
JPBergsma Apr 25, 2023
498f1f5
Merge branch 'JPBergsma_add_meta' of https://github.com/JPBergsma/OPT…
JPBergsma Apr 25, 2023
6cfe92f
Moved mention that suffix '_meta' should not be used for regular fiel…
JPBergsma Apr 25, 2023
1414c55
Add extra single quotes around the '_meta' suffix.
JPBergsma Apr 26, 2023
81f2f96
Added example of a property definition for a metadata field.
JPBergsma Apr 26, 2023
b69b0e8
Update optimade.rst
JPBergsma May 1, 2023
478a572
Clarifined nested meta data fields a bit.
JPBergsma May 1, 2023
d3b7050
Merge branch 'JPBergsma_add_meta' of https://github.com/JPBergsma/OPT…
JPBergsma May 1, 2023
8ed79b5
Placed meta_data_for field in the property definition.
JPBergsma May 1, 2023
f1e179c
Removed property definition metadata_for field as it is now part of t…
JPBergsma May 4, 2023
a60eb5c
Added comma to improve readability.
JPBergsma May 4, 2023
df40172
Added comma to improve readability.
JPBergsma May 4, 2023
d16e97c
Merge branch 'JPBergsma_add_meta' of https://github.com/JPBergsma/OPT…
JPBergsma May 4, 2023
1e81988
Postponed decision on whether or not nested metadata fields are allowed
JPBergsma May 4, 2023
6f56a17
Moved the metadata fields to the JSON API per entry meta field as sug…
JPBergsma Jun 1, 2023
97a0435
Removed x-optimade-metadata-for from example property declaration as …
JPBergsma Jun 1, 2023
00a7a08
Tried to make it clearer when a server SHOULD return the meta data.
JPBergsma Jun 1, 2023
7099a69
removed extra '.'
JPBergsma Jun 2, 2023
09fffea
Apply suggestions from review
rartino Jun 8, 2023
cceabb5
Fix where metadata properties appear based on discussion
rartino Jun 8, 2023
45b98b8
Remove the repetition of info about the data->meta field
rartino Jun 8, 2023
b95c465
Provide an example that is less objectable for metadata
rartino Jun 11, 2023
134a906
Fix examples as requested by workshop
rartino Jun 11, 2023
078f174
Apply suggestions from code review
JPBergsma Jun 12, 2023
e15aaff
Apply suggestions from review
rartino Jun 12, 2023
9c4417f
Fix JSON example spelling error
rartino Jun 12, 2023
b36c372
Update optimade.rst
JPBergsma Jun 14, 2023
cc42d8f
Apply suggestions from review
rartino Jun 14, 2023
52ad0c0
Remove global per-entry metadata properties
rartino Jun 14, 2023
62f987d
Remove ability for per-entry global metadata
rartino Jun 14, 2023
aaddc82
Apply suggestions from review
rartino Jun 15, 2023
656e1fb
Merged upstream develop
rartino Jun 16, 2023
4feb8e1
Update optimade.rst
merkys Jun 16, 2023
6992038
Cleanup trailing whitespace.
merkys Jun 16, 2023
6ffe550
Apply suggestions from review
rartino Jun 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 80 additions & 1 deletion optimade.rst
Original file line number Diff line number Diff line change
Expand Up @@ -492,6 +492,80 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo
// ...
}

Metadata properties
-------------------
rartino marked this conversation as resolved.
Show resolved Hide resolved

A metadata property represents entry and property-specific metadata for a given entry.
How these are communicated in the response depends on the response format.
For the JSON response format, the metadata properties are stored in the resource object metadata field, :field:`meta` in a dictionary field :field:`property_metadata` with the keys equal to the names of the respective properties for which metadata is available, see `JSON Response Schema: Common Fields`_.

The format of the metadata property is specified by the field :field:`x-optimade-metadata-definition` in the Property Definition of the field, see `Property Definitions`_.
Database providers are allowed to define their own metadata properties in :field:`x-optimade-metadata-definition`, but they MUST use the database-provider-specific prefix even for metadata of database-specific fields.
For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`.
The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields.

Implementation of the :field:`meta` field is OPTIONAL.
However, when an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all fields present in the data part of the response which has metadata.

Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`element_ratios` and the database-specific per entry metadata field :field:`_exmpl_originates_from_project` :

.. code:: jsonc
{
"data": [
{
"type": "structures",
"id": "example.db:structs:0001",
"attributes": {
"element_ratios":[0.33336, 0.22229, 0.44425]
},
"meta": {
"property_metadata": {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"property_metadata": {
"attributes_metadata": {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sauliusg @giovannipizzi Right, what did we end up saying about this name?

I was somewhat in favor of "property_metadata" since this is the metadata channel for metadata of OPTIMADE properties when implemented in the JSON response format. I suppose one can argue that OPTIMADE properties are implemented in the JSON response format as attributes (except for id, type...) and as such, naming the channel "attributes_metadata" also makes some sense.

Any thought on this? Do we care at all? I'm assuming @JPBergsma prefers "attributes_metadata" since he proposes the change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer "property" as this is the term used in OPTIMADE specification. Term "attributes" is specific to JSON:API, thus I would suggest us not to tie with it. If in future alongside JSON:API we introduce XML representation with OPTIMADE terms, having both "properties" and "attributes" may add unnecessary confusion.

Copy link
Contributor Author

@JPBergsma JPBergsma Jun 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I did not realize this suggestion was already posted.
During the discussion in the coffee room, we decided that the name would be "attributes_metadata".
In one of the examples on the white boards for the ranged properties, the term "property_metadata" was used, which is probably where @rartino got it from. To me, "attributes_metadata" sounds more fitting for the JSON response, but I do not have a strong preference for the name. If the majority wants "property_metadata", that's also fine with me.

"element_ratios": {
"_exmpl_originates_from_project": "piezoelectic_perovskites"
}
}
}
},
{
"type": "structures",
"id": "example.db:structs:1234",
"attributes": {
"element_ratios":[0.5, 0.5]
},
"meta": {
"property_metadata":{
"element_ratios": {
"_exmpl_originates_from_project": "ferroelectric_binaries"
}
}
}
rartino marked this conversation as resolved.
Show resolved Hide resolved
}
//...
]
// ...
}

Example of the corresponding metadata property definition contained in the field :field:`x-optimade-metadata-definition` which is placed in the property definition of :field:`element_ratios`:

.. code:: jsonc
// ...
"x-optimade-metadata-definition": {
"title": "Metadata for the element_ratios field",
"description": "This field contains the per-entry metadata for the element_ratios field.",
"x-optimade-type": "dictionary",
"x-optimade-unit": "inapplicable",
"type": ["object", "null"],
"properties" : {
"_exmpl_originates_from_project": {
"$id": "https://properties.example.com/v1.2.0/element_ratios_meta/_exmpl_originates_from_project",
"description" : "A string naming the internal example.com project id where this property was added to the database.",
"x-optimade-type": "string",
"x-optimade-unit" : "inapplicable"
}
}
}
// ...

Responses
=========

Expand Down Expand Up @@ -643,7 +717,12 @@ Every response SHOULD contain the following fields, and MUST contain at least :f
- **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON API resource object <http://jsonapi.org/format/1.0/#document-resource-objects>`__ or a *list* of JSON API resource objects.
Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field.

Every resource object MAY also contain a :field:`meta` field with the following keys:
Every resource object MAY also contain a :field:`meta` field which MAY contain the following keys:

- **property_metadata**: an object containing per-entry and per-property metadata.
The keys are the names of the fields in :field:`attributes` for which metadata is available.
The values belonging to these keys are dictionaries containing the relevant metadata fields.
JPBergsma marked this conversation as resolved.
Show resolved Hide resolved
See also `Metadata properties`_

- **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response.
The keys are the names of the fields in :field:`attributes` for which partial data links are available.
Expand Down