DefaultSelectedYN field for Field Resource

[grispin]

Problem: OData assumes all fields for the selected fields of a resource by default and some fields (like cumulative days on market) can be more complex or query performance intensive for the providers to generate that other simple data entry fields. Many feeds do not necessarily need process these fields so the generation effort spent is wasted.

Proposal: Add a Boolean field 'DefaultSelectedYNto theFields` resources to communicate to clients if a field is not part of the query's default wildcard select on a given resource. Fields with the boolean set to False will only be included in the query response payload if they are explicitly listed in the select segment of the query.

[codygustafson]

ExpirationDate, CumulativeDaysOnMarket, and DaysOnMarket are all examples of fields I would like to leverage this for. Being able to ensure that clients have this ability will speed up results for clients that request a default payload but do not need these fields. I do wonder if we can make these fields "expandable" instead of "selectable".

[gr33neggs]

Does odata specify that all fields listed in the metadata for a given resource must return from an open query against said resource?

I could see this being useful to omit fields that aren't applicable to a given property type. But that may lead us down a rabbit hole of property type specific end points.

[grispin]

Odata returns all fields for a resource record by default when select fields are not provided in the query.

This proposal would not work for your use case is conditional on field values with in the record. The metadata (both the edmx and model/field/lookup resource solutions) are at the resource level and have to apply to every record regardless of type. The data shape of the property resource has to be universal to all property types as a result.

[darnjo]

I think this goes against how OData is intended to work?

A $filter statement with no $select statement should always return all possible fields in a given record.

I think this proposal could still be used to inform the client which fields they should use as defaults in their select statements, and they could add more if they wanted, but a request without a $select statement should always include all fields in the payload.

[grispin]

While that would be nice if clients were respectful by using the $select standard to only ask for what fields they are actually using that is not what we see happening today. We see $select on a very small percentage of the actual real transactions and follow up with client show data just being filtered client side. That real world use case triggered this proposal as when clients were asked to use $select little adoption occurred and follow ups with the non-adopters got responses like it is "programmatically easier to filter client side than to extend the work down to the OData query". (client integrator's own words)

Implementations need an in-transport feedback mechanism make the transaction efficient for both parties. This was a first stab for the high cost fields. The current solution of using rate limiting for these kinds of behavours but as we have discussed in previous meetings that approach is a blunt instrument that no one likes and should be a last resort. We should want to find something better. We can always extend the OData behaviours when there is value to RESO like what was done for the Model/Field/Lookup structure to supersede the native OData metadata. OData is merely the foundation for us to build from, not the box within which we must remain. I personally think have the default select-all behaviour was a bad choice by OData.

[darnjo]

I don't think it's reasonable to override the default behavior of OData, whether there seems to be a good reason or not.

We've avoided this kind of choice in the past and should continue to do so going forward.

Once we start coloring outside the lines we're no longer using an open standard. Rather making up our own. This is contrary to the goals of migrating to OData in the first place.

[codygustafson]

I completely agree that we should not break OData with this. I was thinking more in terms of "how can adapt to fit this use case". Whether that means finding an OData Property facet to enable this functionality (doesn't look like one exists), making these fields $expandable (doesn't look possible since navigation properties must be entity types), or possibly making these fields in bound functions instead - I'm not sure if this would work as I expect yet.

[grispin]

I was hoping someone might have a better approach than the solution I proposed. This was just the least worse idea I could come up with.

$expand is not going to work as it would require dividing the resource which would break DD. I don't think property facets can address this. Functions would introduce custom field handling client-side between implementations and requires custom code for each implementation to handle the different field functions in each system. We want to avoid making this happen as we already have custom functions for different implementations within RETS Update and it is terrible to deal with from the client side.

We need to keep this simple or we should just use the tools we already have. If we make it overly complex, it will be a hinderance. By the sound of the discussion, it would be less impactful to/work for the clients and server implementations to use the rate limit approach. The only part I don't like is it leave the communication of how to improve the requests to get less rate-limited external to the transport. This is already a different pain point as it is manual process.

[bryanburgers]

I think this is more specific to DOM/CDOM than your more general approach, but you brought up specifically the cost of calculating DOM/CDOM. (I don't, generally, know what goes into calculating ExpirationDate, either; I'd like to hear about that!)

From the perspective of a data replicator client, I also don't want you to calculate DOM or CDOM. I don't want your calculated DOM/CDOM today because tomorrow it will be out of date.

One partial solution is: we figure out (maybe through the DD group rather than the transport group) what information can be sent from the server to the client that allows the client to determine the current DOM/CDOM.

I feel like I've seen some servers (can't remember specifically) that, instead of sending down DOM (or maybe, in addition to sending down DOM) they send down something like

• Last calculated DOM count
• Date the DOM was last calculated
• Is DOM currently accumulating

On the server, I don't know exactly how or when those get calculated, but I imagine it could be cheaper. And maybe that's already something you're storing to do your calculation?

On the client, we would calculate DOM as

Number of days since calculated = Today - Date the DOM was last calculated

if (Is DOM currently accumulating) {
  DOM = Last Calculated DOM count + 1 * Number of days since calculated
} else {
  DOM = Last Calculated DOM count
}

No suggestion of a solution here, but maybe solving the DOM problem helps solve the original problem?

[codygustafson]

I don't, generally, know what goes into calculating ExpirationDate, either; I'd like to hear about that!

It's not calculating the date - it is calculating the permissions to view it in that case. This is the lightest case of the few mentioned earlier.

[grispin]

CDOM was just the field that I chose to use as the example for the business justification as people readily understand that its behaviour is more complex than simple canned query or cached record which could be more involved. The core of the problem I'm trying to address is the wasted resources of unused data on the server implementations. Clients are requesting data they will never use and implementations spending the resources to generate unused data.

If you take a very small example of an implementation needing an additional 10ms per record to provide the computationally heavy fields with only 1000 records touched per day and only 1000 feeds fetching that record, it translates to ~106 days of compute time over a year for that 10ms increase. The larger vendors are working at a scale that is many orders of magnitude larger than this example so the reductions could be substantial even when the additional per record time is even a single millisecond.

[gr33neggs]

This wording from the OData 4.01 specification sounds like we can actually do this:

If the $select query option is not specified, the service returns the full set of properties or a default set of properties. The default set of properties MUST include all key properties. Services may change the default set of properties returned. This includes returning new properties by default and omitting properties previously returned by default. Clients that rely on specific properties in the response MUST use $select with the required properties or with *.

This sounds like you may return a default set of properties instead of the full set. It seems to give flexibility to determine what that default set is.
The full set isn't assumed to be returned unless $select=* is used.

edit: the next line in the sepc also supports this:

If the service returns less than the full set of properties, either because the client specified a select or because the service returned a subset of properties in the absence of a select, the context URL MUST reflect the set of selected properties and projected expanded navigation properties.

https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358942

v4 has similar text:

If the $select query option is not specified, the service returns the full set of properties or a default set of properties. The default set of properties MUST include all key properties.
If the service returns less than the full set of properties, either because the client specified a select or because the service returned a subset of properties in the absence of a select, the context URL MUST reflect the set of selected properties and expanded navigation properties.

https://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/part1-protocol/odata-v4.0-errata03-os-part1-protocol-complete.html#_Toc453752287

[gr33neggs]

We also need to ensure that odata.context lists out all of the included fields if its not the full set.
i.e. :
{
"@odata.context":"https://example/OData/$metadata#Property(ListingKey,All,The,Other,Fields,In,The,Default,Reponse)",
"value": [
{
...results...
}
]
}

[grispin]

On the HTTP URI limit side, There is nothing specified in the HTTP specification other than server HTTP 414 response code for length restrictions and the limits are open for the implementers to use the error codes to define. Implementations range from 2k to 64k for their defaults but many can be increased (see below for common software defaults).

Browser:
Edge: 2k
Chrome: 32k
Safari & Firefox: 64k
Android: 8k

Servers/CDNs:
Apache: 8k
Microsoft IIS: 16k
AWS Cloudfront : 8k
Cloudflare: 32k

As an example, with the 600 fields and an 8k URI length, that is only ~13 characters for each field name if the entire URI is used for the $select parameter. Add $filter, $expand, etc. that size decreases. Do we need to consider something to expose the query functionality out of the URI and header? (Odata Operation, etc.)

[darnjo]

Thanks for looking up the limits - I guess I was thinking Apache with 8k.

Nginx, also popular, uses 8k by default as well, though it can be increased. Interesting regarding Edge.

There are a lot of opinions about whether it's a good idea or not to have a request body in a GET request, which OData uses for the read operation, but in other scenarios I've used POST with a body for more complex queries.

For bound functions, OData still uses GET requests, so perhaps someone could use a function to combine a predefined select list through a "payload" or "view" with additional computed fields?

GET ~/odata/GetPayloadWithComputedFields("PayloadName", "ComputedField1", "ComputedField2", ..., "ComputedFieldN")

If that seems reasonable, we might also want to be able to pass which resources we want from that payload as well.

OData also supports a POST, which can then have a payload, through bound actions? Maybe a similar approach could be used there? I'm not sure what that would look like though...

The context variable in OData is a JSON string, so it could be as long as it needed, but I guess the assumption is that the user could follow that resolve that context on the server, which would run into the same problems in terms of URI size limits if it were a GET request that had hundreds of fields in a select list.

[gr33neggs]

Do we need to be concerned with URI limits in regards to the odata.context annotation? I know it needs to have the correct syntax, but I am unaware of a situation where the URI it contains is actually used in navigation.

If we want the full set we would simply pass $select=* in the GET

[darnjo]

I think we only need to worry about it if people are expecting to follow the context URI through a GET request?

[darnjo]

Action items for proposal:

• Research whether * actually selects everything (not a problem if not, but we should outline how things behave in the spec)
• Should this be called DefaultSelectedYN or is there an existing OData term that we can use?

[grispin]

Did a bit of research on this.

• * wildcard should to return everything.

It is also possible to request all structural properties, including any dynamic properties, using the star operator. The star operator SHOULD NOT introduce navigation properties, actions or functions not otherwise requested.

• There is nothing in the EDMX or response payload to communicate the field subset list. The client only knows that they have not gotten the full field set because the field count is less than the count in the EDMX definition.

[darnjo]

Thanks much, @grispin. Makes sense.

Edit: Would you mind linking to the section of the OData spec where you found this?

[grispin]

Wildcard: https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_SystemQueryOptionselect
For the second one: There is no direct reference, I had to look at the EDMX and protocol spec to find that nothing was declared.