Replies: 9 comments 39 replies
-
|
It might make sense to define some well-known enumerations in the FeedTypes LookupName, such as IDX and BBO, so they're consistent across markets? It could be open with enumerations so providers could extend it with custom feeds. Proposal looks good, and it addresses requirements raised in meetings in that the solution should be simple and has to be able to filter both at the record (row) and field (column) level. If a given record shouldn't be in an IDX feed anymore, as shown above, IDX would not be in the FeedType array. And if it were, the fields to show in that context would be defined in the Field Resource. |
Beta Was this translation helpful? Give feedback.
-
|
How should we communicate Data Dictionary suggestions for well-known enumerations - there isn't any discussion in Atlassian at the moment. Here's a short list I was given:
|
Beta Was this translation helpful? Give feedback.
-
|
We could have these as examples of different FeedType values but the field should not be limited to those values. We talked during the Feb 2022 meeting, about the value data being an array of strings so that different data provider business use-cases could be represented as required by the data providers. The use of an enum or predefined strings prevents that use-case. An easy example where customized FeedTypes would be needed is two IDX feeds from a single vendor for two different geographic regions. There is also the concern that the FeedType value has some control over the usage of data by the client. The governance of the data usage is controlled by contract between the data provider and the consumer. As we discussed during the call, the agreements with different vendors are all slightly different. The string token is only being used to represent the different feeds during transport by provideing a unique identifier for the feed to that contract. There is no implicit data agreement because the text of FeedType value is "IDX" or "BBO". |
Beta Was this translation helpful? Give feedback.
-
|
@cobogeri - A It would be "Open with Enumerations," meaning that existing definitions exist (and providers should use them where appropriate), but it may be extended with local feed types as well. The In terms of OData "Edm.EnumType" metadata, this would be as follows: <?xml version="1.0" encoding="UTF-8"?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
<edmx:DataServices>
<Schema Namespace="org.reso.metadata" xmlns="http://docs.oasis-open.org/odata/ns/edm">
<EntityType Name="Property">
<Key>
<PropertyRef Name="ListingKey" />
</Key>
<Property Name="ListingKey"
Type="Edm.String" />
<Property Name="ListPrice" Type="Edm.Decimal" />
<Property
Name="FeedType" Type="Collection(org.reso.metadata.enums.FeedTypes)" />
</EntityType>
<EntityType Name="Field">
<Key>
<PropertyRef Name="FieldKey" />
</Key>
<Property Name="FieldKey" Type="Edm.String" />
<Property
Name="FieldName" Type="Edm.String" />
<Property Name="FeedType"
Type="Collection(FeedTypes)" />
</EntityType>
</Schema>
<Schema Namespace="org.reso.metadata.enums" xmlns="http://docs.oasis-open.org/odata/ns/edm">
<EnumType Name="FeedTypes">
<Member Name="IDX" />
<Member Name="BBO" />
<Member Name="VOW" />
</EnumType>
</Schema>
</edmx:DataServices>
</edmx:Edmx>This would be slightly different if String enumerations and the Lookup Resource were used. In that case, it would be as follows: <?xml version="1.0" encoding="UTF-8"?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
<edmx:DataServices>
<Schema Namespace="org.reso.metadata" xmlns="http://docs.oasis-open.org/odata/ns/edm">
<EntityType Name="Property">
<Key>
<PropertyRef Name="ListingKey" />
</Key>
<Property Name="ListingKey"
Type="Edm.String" />
<Property Name="ListPrice" Type="Edm.Decimal" />
<Property Name="FeedType" Type="Collection(Edm.String)">
<Annotation Term="RESO.OData.Metadata.LookupName" String="FeedTypes" />
</Property>
</EntityType>
<EntityType Name="Field">
<Key>
<PropertyRef Name="FieldKey" />
</Key>
<Property Name="FieldKey" Type="Edm.String" />
<Property
Name="FieldName" Type="Edm.String" />
<Property Name="FeedType" Type="Collection(Edm.String)">
<Annotation Term="RESO.OData.Metadata.LookupName" String="FeedTypes" />
</Property>
</EntityType>
<EntityType Name="Lookup">
<Key>
<PropertyRef Name="LookupKey" />
</Key>
<Property Name="LookupKey" Type="Edm.String" Nullable="false" />
<Property Name="LookupName" Type="Edm.String" Nullable="false" />
<Property Name="LookupValue" Type="Edm.String" Nullable="false" />
<Property Name="StandardLookupValue" Type="Edm.String" />
<Property Name="LegacyODataValue" Type="Edm.String" />
<Property Name="ModificationTimestamp" Precision="27" Type="Edm.DateTimeOffset"
Nullable="false" />
</EntityType>
</Schema>
</edmx:DataServices>
</edmx:Edmx>Where the REQUEST RESPONSE {
"@odata.context": "/Lookup?$filter=LookupName eq 'FeedTypes'",
"value": [{
"LookupKey": "L122",
"LookupName": "FeedTypes",
"LookupValue": "IDX",
"StandardLookupValue": "IDX",
"ModificationTimestamp": "2022-07-07T17:36:16Z"
}, {
"LookupKey": "L2358",
"LookupName": "FeedTypes",
"LookupValue": "BBO",
"StandardLookupValue": "BBO",
"ModificationTimestamp": "2022-07-07T17:36:15Z"
}, {
"LookupKey": "L234059",
"LookupName": "FeedTypes",
"LookupValue": "VOW",
"StandardLookupValue": "VOW",
"ModificationTimestamp": "2022-07-07T17:36:14Z"
}]
} |
Beta Was this translation helpful? Give feedback.
-
|
We were preparing to extend our Field Resource like this. Yes, as long as the enumerations aren't closed. And a "yes please and thank you" for some well-known enumerations. We never got to discussing adding "FeedType" to resources. Because naming things is hard. But yeah, that works now we've got it named. |
Beta Was this translation helpful? Give feedback.
-
|
Absolutely we should start with a list of well-known enums: IDX, BBO, VOW, and I'm sure there are more. A business case for this is a data consumer with multiple feeds from a single data source, but each feed has different usage restrictions. This proposal would allow the data consumer to only ingest a single feed, but still have enough information from the data to use records and fields appropriately in the various contexts they are operating in. |
Beta Was this translation helpful? Give feedback.
-
|
On the call we discussed it being an array of strings so that different business use-cases can be represented. The multiple feeds may not always been of different classifications. They may be the same type of feed but for two different regions. The data providers should be able to create custom FeedType values as required to represent the full set of business use-cases. Consumers cannot assume their data usage based on the string value of the FeedType. They need to follow their contracts for the correct data usage restrictions. The FeedType is only a string to represent that contractual agreement, nothing more. If we put meaningful text in the fields it is an approximation of it's usage characteristics but the contract is still the governing document, not the FeedType. |
Beta Was this translation helpful? Give feedback.
-
|
We need to consider the use-cases where a record field visibility is conditional based the values within record and feed data access policies are contradictory on the display of data in that conditional field. Use Cases:
Example: (Completely theoretical) Field Resource presentation must show both feeds as having the feed type because the data could be shown for a record. While there will be identical overlap when the property is Active and deduplication can occur, any other property status will have issues expressing the data access policy for the returned data. The client could interpret this as A possible solution to this would be to send the resource uniquely for each feed in the one query for only those cases where the Field resource cannot expressly define the policy. This does make the response payload larger but allows the data providers to communicate the policy without the need for complex client logic and still keeps the one query solution working. Server side implementations only need to implement this logic if they have need of conditional field access based on record's field value. |
Beta Was this translation helpful? Give feedback.
-
|
This is much cleaner. I do have a question about what would be in the Field Lookup for One other small improvement I would suggest to |
Beta Was this translation helpful? Give feedback.
-
|
These options could definitely meet the need and I'd be curious what others think of this problem/solution. Related to Geoff's question, adding this information to the records does reduce the separation of data and metadata, but that's only an issue if we decide it is. I was going to suggest a solution like this: Paired with an entry in the "Fields" resource that communicates that "PrivateRemarks" is going to sent in this form. We would only need this separation of a field per feed for fields that have this record level conditional logic. Every record in the single feed would have both fields. If the single feed was made up of |
Beta Was this translation helpful? Give feedback.
-
|
I think the Josh's FeedRule solution is a little cleaner for a few reasons.
vs.
|
Beta Was this translation helpful? Give feedback.
-
|
How do others feel about this proposed solution? |
Beta Was this translation helpful? Give feedback.
-
|
There is an open question about how we would represent multiple streams of the same type in the single feed. Multiple BBO or PDAP feeds. Solutions mentioned on the Apr 4th call:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for all of the thoughts on this. With this being a timely, concrete business need expressed by a large portion of membership, we should assess whether we've covered the 90%+ business case with the current proposal. IDX, VOW, and BBO are that business case, with the NAR policy requiring that they be available to a broker participant in a single feed, if requested by the broker. If we have edge cases, let's ensure they are 1) backed up by verifiable examples from our customers and 2) big enough edge cases to warrant additional complexity in the solution for the primary business case. One example edge case is if the feed only provides a field when the resource is in a given part of its lifecycle. What's a specific example here? Are certain fields obscured from data feed consumers based on listing status? Expiration dates (and future closing dates) exist in BBO feeds and should be understood to not include public display usage in any situation. Pending and Closed status dates are included in IDX by default policy, and only obscured in non-disclosure counties and states. The MLS data provider can unilaterally obscure non-displayable data from a data consumer's IDX feed to make it simpler. And if a data consumer needs this simplicity because they can't parse the business rules of IDX/VOW/BBO, they can get an IDX-only feed for display and a separate BBO feed for back office tools. They are not required to get a combined feed, and most MLSs prefer to have multiple options here. This is important to keep in mind. This "one feed" solution, on the other hand, is supposed to make things efficient for the data consumer that can parse a complex feed. The solution may not even need to consider nulling out certain data. If the consumer gets a field's data for any of their feeds, they get the data and they need to follow the rules associated with the feed licenses they've agreed to (and are identified in the FeedType field). If they show inability to properly parse the feed based on IDX/VOW/BBO rules, it seems reasonable that the MLS would move the data consumer back to individual feeds. |
Beta Was this translation helpful? Give feedback.
-
|
As a non-technical person, I’m curious what the differences are between FeedTypes and Payloads. As an MLS, I am giving an entity access to a feed containing one or more payloads. If options such as IDX, VOW, BBO, etc are Payloads and, therefore, outline the approved use of such data, what does the FeedType tell us? I know this matter has been discussed a bit. Can someone clarify the difference for me and the need for having both as opposed to just one? |
Beta Was this translation helpful? Give feedback.
-
|
@jfbreault2022 It's good to fully define these terms, especially because they are used interchangeable a lot of the time. The distinction (in my mind) is that a Payload is which fields are included in a record - for example StandardStatus is part of the IDX payload according to the RESO DD Wiki https://ddwiki.reso.org/display/DDW17/StandardStatus+Field and a feed is a set of records with a given payload. For example the IDX feed may be all active records, all of which are returned with the IDX payload. Most of the time they are interchangeable, but you could have situations where a Payload applies to multiple feeds - maybe there's a VOW feed and BBO feed that have the same payload, but return different sets of records, and are therefore two distinct feeds. |
Beta Was this translation helpful? Give feedback.
-
|
@EnFinlay So in your example of the VOW feed and BBO feed with the same payload but different results, it's that because certain listing statuses, for example, may not be permissible in a VOW feed but are in a BBO, yet the StandardStatus appears in both Payloads? |
Beta Was this translation helpful? Give feedback.
-
|
Yup, that's right. |
Beta Was this translation helpful? Give feedback.
-
|
I am on the MLS where I communicate with brokers and data consumers. For me, data feed is what the data consumer wants from the MLS to power their products. They want a way to access the data. Another possible way to look at it: |
Beta Was this translation helpful? Give feedback.
-
|
So we have multiple vendors that mentioned they have MLS's using field level restrictions in their feeds during the transport meeting today. These were use use cases the mentioned they do it today with separate feeds:
|
Beta Was this translation helpful? Give feedback.
-
|
At Canopy MLS, we suppress data in the following fields before data consumers can see it:
The following will not display when in a pending type status (we have two types): (IDX only)
Only way to get Expired Date is through PDAP (Participant Data Access Policy) payload only. |
Beta Was this translation helpful? Give feedback.
-
|
I think there are two additional questions to address regarding single feeds before we can move this forward? 1. Opting out of a feed on a per-record basisConsider the case where an agent or brokerage wants to opt out of a feed for one or more listings... We have fields to address similar cases at the moment, such as InternetEntireListingDisplayYN, which has a decent amount of usage currently, but does this cover all the cases? Opting out on a per-record basis could be done with the proposed "FeedRule" structure mentioned above, but for the Property Resource this might include hundreds of fields which then impact the payload size. Perhaps we want to consider adding another property like 2. PDAPFor PDAP feeds, there seem to be at least two ways we could potentially address this. There may be others:
In my opinion, it's better to define things explicitly, when possible. There are a couple of ways this could potentially be implemented?
Curious which fields people are using to determine access for PDAP now? Is it just ListOfficeId/Key, etc, or are there other fields used in practice as well? |
Beta Was this translation helpful? Give feedback.
-
|
From where I'm sitting it isn't necessary, but others may disagree. |
Beta Was this translation helpful? Give feedback.
-
|
As an aggregator I would love to grab just a single feed for all of my PDAP clients in a single MLS. The problem being how to make it work as a single feed - The only way I see it is that on each record, there would be a field telling you which officeIDs are allowed to use that record. |
Beta Was this translation helpful? Give feedback.
-
|
If aggregators or other providers working on behalf of participants in each market are currently pulling duplicate data feeds for their participants, it seems to make sense as an optimization to have the ability pull all records once from a given provider (such as an MLS vendor) along with metadata describing the entities that can access a given record. Offices are a good example. Even if this list were long, tens or hundreds of entities, it would be more efficient than pulling records multiple times, with less reprocessing. I'm curious how common this is in practice. As to how it might be implemented, it probably needs further discussion? I think we could potentially split this into two related proposals. One (RCP-035) could cover the additions to the Field Resource and some well-known definitions for feed types. IDX, BBO, etc. The Data Dictionary Workgroup has offered to take this on. It could also include The other could supplement RCP-035 with DRM metadata as to which entities could see a given record. Perhaps there's some other useful information to convey as well. Perhaps we should get the current proposal into DD 2.1 since it's closing, and will help with the PDAP case? Then we can continue to develop the additional features for 2.2 after some further outreach and discussion? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure we need to change anything to support multiple PDAP identities in the single feed case. The login credentials for the single feed would just have unique If the login credentials are attached to two unique PDAP feeds. We would just have |
Beta Was this translation helpful? Give feedback.
-
|
To clarify, I'm looking at it from the point of view of an aggregator who manages many participants and only wants to connect to the MLS/vendor through one set of credentials and one feed to pull information for all their participants. If each PDAP feed is pulled separately at the moment in this scenario (?), with its corresponding credentials provided by the upstream vendors to the providers/aggregators on behalf of the participants, then it seems more efficient for those in this situation to be able pull from one feed with one set of credentials and be able to route things accordingly from there with metadata in the feed. This is probably a fairly limited case at the moment, but does require time/code/resources and credential management for multiple participants can be challenging? I think the PDAP requirements, as currently written, are satisfied by the RCP-035 proposal as is, assuming one set of credentials/feed/feed type per participant. But, things could probably be made more efficient in the many-participant case. That said, if this is something we want to tackle, I think it could be added alongside what's there now as supplemental information in a later proposal. |
Beta Was this translation helpful? Give feedback.
-
|
Progress Update: Discussed on the August 14, 2023 Transport call. The group felt that the proposal with However, it does not address the case where a single vendor is pulling data on behalf of multiple participants within a given MLS. They still would have to pull once per participant/PDAP feed with separate credentials, which results in duplicate data being requested multiple times. The main thing to be addressed at this point is how to convey access information at the entity level, which could have different data attributes in a given case. For example ListOfficeId, ListOfficeKey, etc. One suggestion on the call was to normalize the access information for each feed into its own related record and then use a "custom" FeedType to indicate the access information. For example, maybe |
Beta Was this translation helpful? Give feedback.
-
|
The concern with using a rules based approach is we get in to the business politics around rules (even display rules) and introduce the technical complexity of needing a rules engine or expression language to handle the work. Using a rules approach will make this take longer to get out the door as we will need consensus on a rules solution, that solution would take time to implement and we will see those that still do not want to share rules do something opaque to secure the data without exposing rules they don't want to share. After today's call and the use-cases present, extending the Here is a mock up of the data shape for a We could still have a long list of The pseudo-EDMX would like like this: <Schema Namespace="org.reso.metadata" xmlns="http://docs.oasis-open.org/odata/ns/edm">
<EntityType Name="Feed">
<Key>
<PropertyRef Name="FeedId" Type="Edm.String" />
</Key>
<Property Name="FeedId" Type="Edm.String" />
<Property Name="FeedType" Type="Collection(Edm.String)">
<Annotation Term="RESO.OData.Metadata.LookupName" String="FeedTypes" />
</Property>
<Property Name="RelatedResourceName" Type="Edm.String" Nullable="True" />
<Property Name="RelatedResourceId" Type="Edm.String" Nullable="True"/>
<Property Name="FeedDescription" Type="Edm.String" />
<Property Name="LastModified" Type="Edm.DateTimeOffset" />
</EntityType>
<ComplexType Name="FeedRules">
<Property Name="FeedIds" Type="Collection(Edm.String)" />
<Property Name="FieldNames" Type="Collection(Edm.String)" />
<Property Name="Visible" Type="Edm.Bool" />
</ComplexType>
<EntityType Name="Property">
<Key>
<PropertyRef Name="ListingKey" />
</Key>
<Property Name="ListingKey" Type="Edm.String" />
<Property Name="ListPrice" Type="Edm.Decimal" />
<Property Name="FeedIds" Type="Collection(Edm.String)" />
<Property Name="FeedRules" Type="Collection(org.reso.metadata.FeedRule)">
</EntityType>
<EntityType Name="Field">
<Key>
<PropertyRef Name="FieldKey" />
</Key>
<Property Name="FieldKey" Type="Edm.String" />
<Property Name="FieldName" Type="Edm.String" />
<Property Name="FeedIds" Type="Collection(Edm.String)" />
</EntityType>
<EntityType Name="Model">
<Key>
<PropertyRef Name="ModelKey" />
</Key>
<Property Name="ModelKey" Type="Edm.String" />
<Property Name="ModelName" Type="Edm.String" />
<Property Name="FeedIds" Type="Collection(Edm.String)" />
</EntityType>
</Schema>
<Schema Namespace="org.reso.metadata.enums" xmlns="http://docs.oasis-open.org/odata/ns/edm">
<EnumType Name="FeedTypes">
<Member Name="IDX" />
<Member Name="BBO" />
<Member Name="VOW" />
</EnumType>
</Schema> |
Beta Was this translation helpful? Give feedback.
-
We have talked about this previously and one of the goals/assumptions is that we were avoiding machine-executable rules or grammars and only passing metadata that would allow consumers to make correct decisions based on the data that was present, rather than trying to convey business rules. Edit: Reading what's outlined above, I'm not sure it handles the case of conveying who can see the record (without assuming that it would be implicitly conveyed in the FeedTypes with a custom FeedId)? I also think that RelatedX complicates things a bit and isn't strictly necessary? The goal is to support the following:
This doesn't have to use "business rules," per se. On the call today, and in a past post (Item 2: PDAP), we discussed it being possible to express things implicitly or explicitly, with the latter being my own personal preference. Assumption 1 Assumption 2 Related Questions:
If the assumptions hold, and favoring an explicit approach, I think it would be something like the following? {
"StandardStatus": "Pending",
"PrivateRemarks": "This is an example remark",
"FeedType": ["IDX", "BBO"],
"FeedRule": [
{
"FieldName": "PrivateRemarks",
"FeedType": "IDX",
"Visible": true
}
],
"ParticipantAccessInfo": {
"DataAccessDeterminationFieldKey": "<FieldKeyFromFieldResource>",
"DataAccessEntityIdsOrKeys": [ "12345", "12346"]
}
}I've made the above example extra verbose for clarity, and the names definitely need some more work,...the example is more illustrative than normative.
Now, this is a bit gross/verbose in that all this information might accompany the record, which could bloat the payload quite a lot for those who don't need to worry about this scenario. One option is to perhaps try and encapsulate what we can (combine FeedRule and ParticipantAccessInfo into a single, nested property) and/or expand it in instead, so those with this case can ask for it on demand. Another option is to only show these fields to aggregators who can consume a single feed in this manner. Not a huge fan of yet another sibling property, but one of the goals was to maintain compatibility with the existing proposal. |
Beta Was this translation helpful? Give feedback.
-
|
This item has been approved by the Transport Workgroup as an RCP. Please make any additional comments on the following issue: #96 |
Beta Was this translation helpful? Give feedback.
-
This post is a follow up from RCP 035. The goal of RCP-035 was to propose a new resource that could be used to help meet the needs of the community by including metadata as data (the Usage resource) to communicate the permissions and allowed usage for each field sent to a data consumer. I would like to withdraw that RCP or heavily revise it as proposed below.
Synopsis
A new field called
FeedTypeof type Multi Enum String should be included in every data resource and the fields resource. On data resources, this field will be used by the data provider to indicate which feed included the given record. On the fields resource (which is a metadata resource), this field will be used by the data provider to indicate whether the given field is included in a given feed type.Business Case
A business case for this is a data consumer with multiple feeds from a single data source, but each feed has different usage restrictions. This proposal would allow the data consumer to only ingest a single feed, but still have enough information from the data to use records and fields appropriately in the various contexts they are operating in.
Example
Hypothetically a data consumer could have two feeds from an MLS - IDX and BBO. The IDX feed contains only Active records, and of those records it only includes the
ListPrice,StandardStatusandUnparsedAddressfields. The BBO feed contains only Closed records, and of those records it only includes theListPrice,StandardStatus,UnparsedAddress, andCloseDatefields.Currently the data consumer would have two feeds:
Feed 1:
Feed 2:
This proposal would present the data as a single feed that would look like this:
and the Fields Resource would look like this:
Given this setup, the data consumer knows that every data record with the "IDX" feed type can be used for "IDX" purposes, and that records with the "BBO" feed type can only be used for "BBO" purposes. Additionally, given the requirement that every record has the same schema, the "Fields" resource comunicates that even though the "CloseDate" field is present on an "IDX" record, that field cannot be used, since that record is not part of the "BBO" feed. Data providers could
nullout the value of the fields that the data consumer does not have permission to use.Proposal
New Field on Every Data Resource
A new field called
FeedTypeof type Multi Enum String will denote which feeds a given record is included in. The values of this field will correspond to different feeds that a data consumer is approved for. These feed names do not necessarily need to be human readable, although it's important that data provider and consumer agree on the allowed usage of each feed.New Field in Fields Resource
A new field called
FeedTypeof type Multi Enum String will denote which fields are included in a given feed. This field will have the same lookup values as theFeedTypefield on the data resources.Beta Was this translation helpful? Give feedback.
All reactions