Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tdl 13966 update custom fields #36

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

dbshah1212
Copy link
Contributor

Description of change

  • Update custom fields schema for supported stream.
  • Removed redundant code of custom fields and added $ref

Manual QA steps

  • Run discovery and validated the catalog.json
  • Run sync and checked custom field values for invoices and estimates streams.

Risks

Rollback steps

  • revert this branch


# verify custom field schema
if stream in self.custom_command_streams:
actual_custom_field_keys = list(schema["properties"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be testing this in metadata not annotated-schema. The menagerie method call is misleadingly named. But annotated-schema is no longer used by our front-end, only metadata is. You should be able to make the same assertion just change the way the actual value is accessed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kspeer825 This test is validating nested fields present under the CustomField field.
However, as metadata only contains first-level fields, can you please help us on how to retrieve nested fields from metadata?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure this is possible. The metadata is what drives table and field selection in our ui. And nested field selection is not a feature that I am aware of, so I think this would require a change to singer as well as our frontend. I will confirm this with a developer and get back to you.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed this with the devs. I think in this test discovering the top level key from metadata is sufficient. If you need to confirm that the subfields are picked up by the tap, you would need a test that performs a sync and you could pull the actual field values as well as the json schema from the messages sent to the target.

Copy link
Contributor

@savan-chovatiya savan-chovatiya Sep 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kspeer825, Updated test, and accessed top-level field value from metadata instead of annotated-schema. The subfields validation from sync is already added as part of the test_quickbooks_sync_all.py test.
NOTE: The CircleCI build is failing currently due to dependency installation failure which is observed in some taps from last week.

Copy link
Contributor

@kspeer825 kspeer825 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests look good. For other taps experiencing this dependency issue, we have unpinned ipdb in setup and split up the install groups into a dev and `test. Here is an example of the change singer-io/tap-google-analytics@a7cbd77 .

@savan-chovatiya
Copy link
Contributor

Tests look good. For other taps experiencing this dependency issue, we have unpinned ipdb in setup and split up the install groups into a dev and `test. Here is an example of the change singer-io/tap-google-analytics@a7cbd77 .

Updated setup to resolve dependency issue as per above reference.

@savan-chovatiya savan-chovatiya requested review from luandy64 and zachharris1 and removed request for KAllan357 September 22, 2021 12:44
dbshah1212 and others added 2 commits October 6, 2021 11:21
* TDL-13967: Added ProfitAndLoss report stream

* TDL-13967: Handled state updation if no records found

* Added comments in unittests

* Updated sync function for report streams

* Fixed integration tests

* Added stream into readme

* Break down report parser for rows and columns

* Resolved circleci failure

* Updated date retrival from datetime object

* Break down dev and test ddependancy in setup

* TDL-13967: Resolved review comments

* TDL-13967: Added code comments

Co-authored-by: savan-chovatiya <[email protected]>
Co-authored-by: Kyle Allan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants