Replies: 7 comments 12 replies
-
|
Is it possible to know the language the default "PublicRemarks" field is in? It looks like the best you can do is assume as a data consumer. What do you think about duplicating the english translation in the Translations field? I'm curious why we don't want data dictionary changes. It seems alright to change the type of PublicRemarks to be some sort of RESO-defined multi-lingual text field. What if we changed PublicRemarks to be a complex type or array of complex type? Or perhaps: |
Beta Was this translation helpful? Give feedback.
-
|
In the proposal above, the assumption is that the payload is in the first language requested by the consumer if the server supports it, and in the default language if they don't. This is intended to match the content negotiation behavior. I think it would be reasonable to repeat the translation for a given field in the Translations "object." The reason I didn't was to keep the payload small. If there are a lot of translated fields, it could add a lot to the record size, especially in cases like PublicRemarks where there's lots of text. While we could redefine fields such as PublicRemarks to be complex types, the goal was to keep the payloads consistent with the current specifications so we don't introduce breaking changes.
PublicRemarks is currently used in 87% of systems and 95% of organizations. Changing the definition would be a major change. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, it would be a major change. This translations approach seems like we will be going towards a future where the standard fields no longer exist for these free text fields. Instead they would all be accessed through Translations. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe we can make it work more cleanly?
We should talk.
…On Mon., Oct. 31, 2022, 8:18 a.m. Cody Gustafson, ***@***.***> wrote:
Yeah, it would be a major change. This translations approach seems like we
will be going towards a future where the standard fields no longer exist
for these free text fields. Instead they would all be accessed through
Translations.
—
Reply to this email directly, view it on GitHub
<#37 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AM7WR7FIEDUHAUBYMCIIS4DWF7PMHANCNFSM6AAAAAAQ4KRZ5U>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Where does the below requirement come from?
I think this would be an easier problem to tackle if we allowed for one request per translation. |
Beta Was this translation helpful? Give feedback.
-
|
We talked about making one request for each translation but the group asked that we have one way to fetch all the relevant translations without the additional requests. One translation per request could still be implemented this way. The client could ask for one at a time and it would still work. More of an optimization. |
Beta Was this translation helpful? Give feedback.
-
|
Feels like another instance of the tension between replication and live queries. |
Beta Was this translation helpful? Give feedback.
-
|
I think that's an accurate statement. For live queries, one would only usually need one translation at a time. For replication, a consumer might want to pull all representations matching the locales they're interested in. The proposal supports either case. |
Beta Was this translation helpful? Give feedback.
-
|
Trying to think of ways to make this more palatable, less implicit, and use more conventional OData paradigms... Issue 1: Headers and Content NegotiationEven though we want to use existing content negotiation standards, when possible, having the additional Translations show up implicitly based on what's in the headers goes against the grain a bit. It seems reasonable to allow the headers to set the primary language but not use them to request additional ones. Changing primary languages is an issue we need to address anyway, so using existing content negotiation standards makes sense for that particular case. Existing fields would be used in this case and existing Data Dictionary models could be preserved. Issue 2: Translations Field as a Complex TypeWhile it would be nice to use more lightweight, non-relational data shapes as models, one issue with doing so in this case is that the Translations field would always be present if it were created as a Complex Type. It should probably be more "on demand." OData would also expect users to filter the values in the field rather than having it be empty by default and only showing values when requested. This would make things a bit irregular. Option 2: Translation Resource with Expansion Instead of Complex TypeAnother approach would be to create a Translation Resource with a "belongs to" relationship similar to Media. The records in Translation would know which items they belong to and many resources and records could share the Translation resource to keep the number of resources to a minimum. Unlike a Complex Type, the Translation Resource could be "expanded in," as needed. <EntityType Name="Translation">
<Property Name="TranslationKey" Nullable="false" Type="Edm.String" />
<Property Name="ResourceName" Nullable="false" Type="Edm.String" />
<Property Name="ResourceRecordKey" Nullable="false" Type="Edm.String" />
<Property Name="Locale" Nullable="false" Type="Edm.String" />
<!-- If we wanted to be strict here and normalize things, we could use FieldKey -->
<Property Name="FieldName" Nullable="false" Type="Edm.String" />
<Property Name="Value" Nullable="false" Type="Edm.String" />
</EntityType>
<EntityType Name="Property">
...
<Property Name="PublicRemarks" Type="Edm.String" />
<NavigationProperty Name="Translations" Type="Collection(Translation)" />
...
</EntityType>In terms of use, nothing would change from how things are currently, which is one of the goals of this proposal. If the consumer wanted a single translation in another locale, they could use content negotiation: Request Response However, if they wanted other translations, they could expand the Translations in and filter by the additional translations they want, for example en-CA: Request Response Pros
Cons
Some options to avoid the "Cons" would be:
Option 3: Translation Resource with Inner Complex Type for TranslationsIn order to make the Translation Resource less vertical, a Complex Type can be used to model all translations for a given record and locale. <ComplexType Name="TranslationValue">
<Property Name="FieldName" Nullable="false" Type="Edm.String" />
<Property Name="Value" Nullable="false" Type="Edm.String" />
</ComplexType>
<EntityType Name="Translation">
<Property Name="TranslationKey" Nullable="false" Type="Edm.String" />
<Property Name="ResourceName" Nullable="false" Type="Edm.String" />
<Property Name="ResourceRecordKey" Nullable="false" Type="Edm.String" />
<Property Name="Locale" Nullable="false" Type="Edm.String" />
<Property Name="Values" Nullable="false" Type="Collection(TranslationValue)"
</EntityType>
<EntityType Name="Property">
...
<Property Name="PublicRemarks" Type="Edm.String" />
<NavigationProperty Name="Translations" Type="Collection(Translation)" />
...
</EntityType>Querying for multiple translations in this case would have a similar syntax as the previous example. Request Response The difference in this case is that there is a set of nested values for each locale. This remedies the extreme verticality of the Translation Resource shown in Option 2, above. One thing that's a bit harder with this approach is selecting specific fields in the Translation since doing so requires a nested Let's say the consumer only wants the PublicRemarks field. Request Response One saving grace is that this kind of query would likely be an edge case. The common cases are:
|
Beta Was this translation helpful? Give feedback.
-
|
I believe the hardest piece of this change is trying to address the different needs between a replication case and a real-time query case. Both are valid use cases but we have been heavily focused on the replication case since that is the more prevalent usage. The real-time case will prefer content negotiation with human readable names in the data payload - the less work for the client the better. The replication case will prefer no content negotiation with all the translations in a single payload. So, we either choose one of these cases, make an endpoint that caters to both, or somehow create a different view or presentation of the data depending on what the client's use case is. If we "choose one of these cases", I expect it will be the replication case. Then, I would expect no content negotiation and we should change the data shape to include all the translations in a meaningful way. If we "make an endpoint that caters to both", we will end up with something akin to the proposal we have here. In this case, content negotiation affects some of the fields but not other. Replicators will use the fields which it does not affect and will have to make changes to leverage those. Real time requests should likely use content negotiation but would have a choice to handle that on their end as well using the "replication style fields". If we chose the real time case, then replicators will be asked to pull data multiple times for every language. If we "create a different view", we will likely have something akin to a "PropertyForReplication" endpoint or something similar for every entity type, where the view (endpoint in this case) is geared to a specific use case. |
Beta Was this translation helpful? Give feedback.
-
|
In Option 3, above, the client who replicates can pull the translations for the locales they want by expanding Translations into the payload and filtering on them in a natural way. This allows them to ask for translations on demand rather than obtaining them by default (as the field was a Complex Type initially in the post, meaning it would always appear). There might be some duplication of data between the values in Translations and those in the payload, but if someone is concerned about that they can combine the headers and filtering by Translations to get back exactly what they need. The main goal was to not have to make additional requests to pull multiple individual translations and I think this addresses that. The live query case would likely only use the single translation, which is possible too. In either case, if the client omitted the |
Beta Was this translation helpful? Give feedback.
-
|
Regarding the December 5, 2022 Transport Meeting discussion, there was a question as to why the translations for a given resource didn't have a more declarative and static structure. For example, Property would have a PropertyTranslation resource that contains the set of fields that could have possible translations, which could be expanded into the Property Resource as a set of Translations, each with their own locale. This is for the following reasons:
|
Beta Was this translation helpful? Give feedback.
-
It's not a 'bad' thing, it's just that the information is already conveyed with the schema. The MaxLength that's in a given existing field, for example PublicRemarks, would have to be the max of the lengths of the data in that field (including translations) since in the single-translation case the consumer could request the translation and have it come out in the given field in the payload.
Let's assume a provider supports Spanish translations on two fields for the Property Resource initially, one standard and one local.
What this is saying is, the number of translated fields in a given static schema, like PropertyTranslation, is going to be the max number of translations a given record has. So, if one record has 25 translations and most have 3-4 on average, then the schema of the PropertyTranslation Resource would have to include all of the possible translated fields for each record. One assumption is that the expected value in the example tends to not be translated. If, on the other hand, the common case is that 24 of those 25 fields are filled in for each translation in every record, then the additional nulls are less of an issue. I'd be curious to hear stats from providers regarding how many translated values are filled in when they offer them. What is the average number of translated items that have values in them out of the the total possible?
While it's similar in that they "process metadata," it adds to the amount of complexity needed to process a single resource like Property. Especially if there's metadata spread out among OData XML Metadata and the Field resource (which isn't required). Now the user needs to a) determine if there are translations on that resource by looking either at the Field metadata (if present) or the expansions for that resource to see if there's a "Translations" relationship, and b) then do introspection on the underlying model (or Field Resource data) to extract which values are translated for each translated model, PropertyTranslation, MemberTranslation, etc. If there were translations for every model in the Data Dictionary, this doubles the number of resources in the Data Dictionary (and in the Field metadata) and requires that the consumer always be in sync with those declarative schema when metadata changes (say a new translated field is added). In other words, it places a lot of dependencies on interpreting metadata in a case that doesn't seem to require it. We already have the payload shape, in the Property Resource, which is the important static structure here. Why do we need a static shape for a collection of translated string fields that are already defined in the payload in order to bulk-transfer them for the replication case? Sending the translations as a collection of dynamic key/value pairs in the payload allows the consumer to understand the translations with zero dependencies on server metadata. Consider cases where a consumer picks up the data from an S3 bucket or external API, for example, using RCP-025 and Data Dictionary JSON Payloads. The provider may not have any OData Metadata or Field Resource at that point, but the consumer could understand the data set from the data alone rather than having an additional metadata dependency.
In this case, static structure provides less predictability of the schema across markets. Now schema can vary between providers and resources depending on the accompanying PropertyTranslation or MemberTranslation fields that are defined in a given market. There could be 500 different PropertyTranslation resources, for example, one for each market. Having a simple collection of Having two known fields is therefore more predictable than 500 PropertyTranslation resources, 500 MemberTranslation resources, etc., and having to do introspection on each one. |
Beta Was this translation helpful? Give feedback.
-
|
I haven't had a chance to read everything written above, but I want to respond to this bit:
Models with many null values hasn't bothered us in the past; I don't think it should be a top level concern for this work.
I think this is a reason to use something like |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for all the feedback, Josh. I believe we have a mutual understanding. I'll just highlight one more point with the MaxLength: This might be a hypothetical where differing languages don't have different lengths, but I feel like in most software systems, they do. If we don't have max-lengths based on language then we'll have spontaneous errors when attempting to insert strings if some languages provide lengths less than the advertised max. I'm not saying this alone is a deal-breaker. We can say all languages must have the same characteristic as their metadata equivalent. The less we use statically defined attributes, the less we can lean on OData for the problems we run into. Right now, I'm feeling like the main benefit to provide a translations entity is to expedite getting something into the standard. |
Beta Was this translation helpful? Give feedback.
-
|
Responding to @EnFinlay:
I feel like it's the opposite - that forcing this to happen through IMO, it's much simpler to have a key/value pair that's predictable across markets rather than yet another set of metadata that needs introspection. The data structure itself enforces the contract rather than needing to rely on the client to interpret metadata. Also, consider the case above where translations for fields are added to or removed from a given resource in some markets but not in others. If fixed schema are used, then knowledge of each of these distinct schemas needs to be stored and accounted for, and it's a major (unnecessary) conversion each time. For what? Just to know that I have another possible string value for If a dynamic list of translations is used, then the consumer only needs to reprocess any records that have changed and this is easy for them to do without changing their existing data structures or maintaining different ones from market to market. The main use case here is to optimize bulk-transfer for replication. As such, I would also argue that having potentially hundreds of empty fields for each additional translation for a given record, when only a handful is needed, is undesirable. It results in additional bandwidth and processing that adds up over time. Live queries will use the translated values in the existing shape for each resource being translated. If a French Canadian translation is being provided for a listing and it has |
Beta Was this translation helpful? Give feedback.
-
|
@codygustafson, responding to this:
I don't think it's hypothetical, but I do think it's already addressed by the proposal. For the client who only needs one translation (e.g. live query case), the goal is that they could request a translated version of the top level resource and get it in the current schema. Property, for example. As such, PublicRemarks and any other translated fields would have to be declared in such a way that their length is the max length of all translations so their data may be passed through the same field. |
Beta Was this translation helpful? Give feedback.
-
|
Beyond the technical pros/cons of relational structure vs. a collection of key/value pairs, there's a larger question here: In the conversations thus far, those suggesting more of a relational approach seem to want the Data Dictionary take on a prescriptive, normalized structure based on MVC principles. While this may be convenient for those implementing their systems that way, it's an opinionated approach that may limit or even preclude other ways of doing things now or in the future. The Data Dictionary is more of an generic interface consisting of coarse buckets and shapes. It's not intended to be an idealized relational database schema for use in people's implementations, nor does it have to be made into one. Providers should be able to implement their systems however they want under the hood and not be forced into a particular style of thinking, and the dictionary should avoid taking on implementation details, like database normalization, in its definition. As the saying goes, "program to the interface, not the implementation." Loosely coupled systems are more flexible and easier to change or improve behind the scenes without causing breakage, and the Data Dictionary should be able to adapt to other environments and technologies. |
Beta Was this translation helpful? Give feedback.
-
Summary
In order to provide support for multiple locales, there is a need to be able to convey one or more translations in Data Dictionary payloads for use with JSON Web APIs.
Requirements
Proposal
A new ComplexType called
Translationcan be created to provide additional translations.It would be defined in OData XML Metadata as follows:
When a consumer makes a request, if they are only looking for a single locale then the existing field in the Data Dictionary would be used to transport the value. If multiple locales are requested and supported by the server, the primary payload would contain information for the first locale, and the
Translationsarray would contain a list of additional locale specific values for that payload.This can be done using the Content Language request and response header, which can both contain multiple locales.
Example 1
Assume a server only supports a locale of
en-US. If a client requests languages besides the one the server supports, the response would contain aContent-Language: en-USheader, and the values would be in Data Dictionary format:REQUEST
RESPONSE
Example 2
Given a server that supports both Canadian English and French, the client requests both, with English as the first preference.
REQUEST
RESPONSE
The primary payload consists of the client's preferred translation, which achieves the goal of keeping the existing Data Dictionary payload structure consistent with what it is currently, and the
Translationsarray contains an additional list of zero-to-many locales and values the provider supports.If the server supported no additional translations, the response would contain the primary locale and the
Translationsarray would be empty. This allows consumers to know whether they need to parse additional translations by checking for a non-zero array, which is convenient.If additional translations are present, as shown above, the consumer can retrieve them in a single request, without requiring that additional language-specific fields be added to the Data Dictionary, e.g. PublicRemarks_en_US.
Example 3
This approach also works for Add/Edit without the need for locale-specific fields to be added to the Data Dictionary.
Assume a data producer is trying to add a new record to a given system.
REQUEST
RESPONSE
Another benefit is that it abstracts the transport of translations from their underlying implementation, meaning providers can store additional translations in whatever way makes sense for their system.
Example 4
In order to query for locale specific values, the client would specify the locale code they want to query by first, with additional ones being optional, and the query would contain a value they want to search for.
For example, to search PublicRemarks for 'favori' (
fr-CA) and also retrieve the translation foren-CA, the request would be as follows:REQUEST
RESPONSE
Beta Was this translation helpful? Give feedback.
All reactions