TDL-22338 add new fields, streams and bug fixes (#7)

* add account_balance stream * adds account_balance stream * add account_sid field to keys stream * change query params for usage_records and alerts and pagination_url for alerts stream * next_page url * revert bookmarking >= and fallback to default value for unset config values * Tdl 22160 add integration tests (#6) * setup circleci * Update config.yml * fix community issues * fix pagination issue * update query params for calls stream * updated replication_key and query_param for calls stream * add basetest for integration tests * remove replication_keys for full_table stream accounts * changelog and version bump * update changelog * remove foreign keys * add discovery test * add pagination test * add all fields and automatic fields * deleted old tests * deleted old test and removed extra lines in circleci config * add start_date and bookmark test * PR comments * update comments * PR comments * update bookmarks test * update pagination test * changelog and version update * update README
singer-io · Mar 20, 2023 · fe70da7 · fe70da7
1 parent 7fd3fb0
commit fe70da7
Show file tree

Hide file tree

Showing 21 changed files with 1,221 additions and 203 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -50,4 +50,4 @@ workflows:
                 - master
     jobs:
       - build:
-          context: circleci-user
+          context: circleci-user
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,16 @@
 # Changelog
 
+## 0.1.0
+  * Fixes following issues [#7](https://github.com/singer-io/tap-twilio/pull/7)
+      * Adds `emergency_address_status` field to `incoming_phone_numbers` stream 
+      * Adds `label` field to `conference_participants` stream
+      * Adds `reason_conference_ended` and `call_sid_ending_conference` fields to `conferences` stream
+      * Adds `price_unit` field to `messages` stream
+      * Adds `account_sid` field to `keys` stream during process_records
+      * Adds `account_balance` stream implementation
+      * Updates/reverts query_params for `alerts` and `usage_records` streams
+      * Fixes pagination issue for `alerts` issue
+
 ## 0.0.3
   * Fixes following issues [#4](https://github.com/singer-io/tap-twilio/pull/4)
       * Fixes pagination issue

diff --git a/README.md b/README.md
@@ -8,27 +8,29 @@ This tap:
 
 - Pulls raw data from the [twilio Advertiser API]([xxx](https://support.twilio.com/s/advertiser-api-documentation))
 - Extracts the following resources:
- - [accounts](https://www.twilio.com/docs/usage/api/account#read-multiple-account-resources)
- - [addresses](https://www.twilio.com/docs/usage/api/address#read-multiple-address-resources)
- - [dependent_phone_numbers](https://www.twilio.com/docs/usage/api/address?code-sample=code-list-dependent-pns-subresources&code-language=curl&code-sdk-version=json#instance-subresources)
- - [applications](https://www.twilio.com/docs/usage/api/applications#read-multiple-application-resources)
- - [available_phone_number_countries](https://www.twilio.com/docs/phone-numbers/api/availablephonenumber-resource#read-a-list-of-countries)
- - [available_phone_numbers_local](https://www.twilio.com/docs/phone-numbers/api/availablephonenumberlocal-resource#read-multiple-availablephonenumberlocal-resources)
- - [available_phone_numbers_mobile](https://www.twilio.com/docs/phone-numbers/api/availablephonenumber-mobile-resource#read-multiple-availablephonenumbermobile-resources)
- - [available_phone_numbers_toll_free](https://www.twilio.com/docs/phone-numbers/api/availablephonenumber-tollfree-resource#read-multiple-availablephonenumbertollfree-resources)
- - [incoming_phone_numbers](https://www.twilio.com/docs/phone-numbers/api/incomingphonenumber-resource#read-multiple-incomingphonenumber-resources)
- - [keys](https://www.twilio.com/docs/usage/api/keys#read-a-key-resource)
- - [calls](https://www.twilio.com/docs/sms/api/message-resource#read-multiple-message-resources)
- - [conferences](https://www.twilio.com/docs/voice/api/conference-resource#read-multiple-conference-resources)
- - [conference_participants](https://www.twilio.com/docs/voice/api/conference-participant-resource#read-multiple-participant-resources)
- - [outgoing_caller_ids](https://www.twilio.com/docs/voice/api/outgoing-caller-ids#outgoingcallerids-list-resource)
- - [recordings](https://www.twilio.com/docs/voice/api/recording#read-multiple-recording-resources)
- - [transcriptions](https://www.twilio.com/docs/voice/api/recording-transcription?code-sample=code-read-list-all-transcriptions&code-language=curl&code-sdk-version=json#read-multiple-transcription-resources)
- - [queues](https://www.twilio.com/docs/voice/api/queue-resource#read-multiple-queue-resources)
- - [message_media](https://www.twilio.com/docs/sms/api/media-resource#read-multiple-media-resources)
- - [usage_records](https://www.twilio.com/docs/usage/api/usage-record#read-multiple-usagerecord-resources)
- - [usage_triggers](https://www.twilio.com/docs/usage/api/usage-trigger#read-multiple-usagetrigger-resources)
- - [alerts](https://www.twilio.com/docs/usage/monitor-alert#read-multiple-alert-resources) 
+- [accounts](https://www.twilio.com/docs/usage/api/account#read-multiple-account-resources)
+  - [account_balance](https://www.twilio.com/docs/usage/api/account#read-multiple-account-resources)
+  - [addresses](https://www.twilio.com/docs/usage/api/address#read-multiple-address-resources)
+    - [dependent_phone_numbers](https://www.twilio.com/docs/usage/api/address?code-sample=code-list-dependent-pns-subresources&code-language=curl&code-sdk-version=json#instance-subresources)
+  - [applications](https://www.twilio.com/docs/usage/api/applications#read-multiple-application-resources)
+  - [available_phone_number_countries](https://www.twilio.com/docs/phone-numbers/api/availablephonenumber-resource#read-a-list-of-countries)
+    - [available_phone_numbers_local](https://www.twilio.com/docs/phone-numbers/api/availablephonenumberlocal-resource#read-multiple-availablephonenumberlocal-resources)
+    - [available_phone_numbers_mobile](https://www.twilio.com/docs/phone-numbers/api/availablephonenumber-mobile-resource#read-multiple-availablephonenumbermobile-resources)
+    - [available_phone_numbers_toll_free](https://www.twilio.com/docs/phone-numbers/api/availablephonenumber-tollfree-resource#read-multiple-availablephonenumbertollfree-resources)
+  - [incoming_phone_numbers](https://www.twilio.com/docs/phone-numbers/api/incomingphonenumber-resource#read-multiple-incomingphonenumber-resources)
+  - [keys](https://www.twilio.com/docs/usage/api/keys#read-a-key-resource)
+  - [calls](https://www.twilio.com/docs/voice/api/call-resource#read-multiple-call-resources)
+  - [conferences](https://www.twilio.com/docs/voice/api/conference-resource#read-multiple-conference-resources)
+    - [conference_participants](https://www.twilio.com/docs/voice/api/conference-participant-resource#read-multiple-participant-resources)
+  - [outgoing_caller_ids](https://www.twilio.com/docs/voice/api/outgoing-caller-ids#outgoingcallerids-list-resource)
+  - [recordings](https://www.twilio.com/docs/voice/api/recording#read-multiple-recording-resources)
+  - [transcriptions](https://www.twilio.com/docs/voice/api/recording-transcription?code-sample=code-read-list-all-transcriptions&code-language=curl&code-sdk-version=json#read-multiple-transcription-resources)
+  - [queues](https://www.twilio.com/docs/voice/api/queue-resource#read-multiple-queue-resources)
+  - [messages](https://www.twilio.com/docs/sms/api/message-resource#read-multiple-message-resources)
+    - [message_media](https://www.twilio.com/docs/sms/api/media-resource#read-multiple-media-resources)
+  - [usage_records](https://www.twilio.com/docs/usage/api/usage-record#read-multiple-usagerecord-resources)
+  - [usage_triggers](https://www.twilio.com/docs/usage/api/usage-trigger#read-multiple-usagetrigger-resources)
+- [alerts](https://www.twilio.com/docs/usage/monitor-alert#read-multiple-alert-resources) 
 
 - Outputs the schema for each resource
 - Incrementally pulls data based on the input state
@@ -45,9 +47,17 @@ This tap:
 - Transformations: subresources_to_array
 
 
+[account_balance](https://www.twilio.com/docs/usage/api/account#read-multiple-account-resources)
+- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Balance.json
+- Parent: accounts
+- Primary key fields: account_sid
+- Replication strategy: FULL_TABLE
+- Transformations: None
+
+
 [addresses](https://www.twilio.com/docs/usage/api/address#read-multiple-address-resources)
 - Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Addresses.json
-- Parent: account
+- Parent: accounts
 - Primary key fields: sid
 - Replication strategy: INCREMENTAL
 - Transformations: subresources_to_array
@@ -116,6 +126,7 @@ This tap:
 - Replication strategy: INCREMENTAL
 - Transformations: subresources_to_array
 
+
 [calls](https://www.twilio.com/docs/sms/api/message-resource#read-multiple-message-resources)
 - Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Calls.json
 - Parent: accounts
@@ -172,6 +183,14 @@ This tap:
 - Transformations: subresources_to_array
 
 
+[messages](https://www.twilio.com/docs/sms/api/message-resource#read-multiple-message-resources)
+- Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages.json
+- Parent: accounts
+- Primary key fields: sid
+- Replication strategy: INCREMENTAL
+- Transformations: subresources_to_array
+
+
 [message_media](https://www.twilio.com/docs/sms/api/media-resource#read-multiple-media-resources)
 - Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages/{ParentId}/Media.json
 - Parent: messages

diff --git a/setup.py b/setup.py
@@ -3,7 +3,7 @@
 from setuptools import setup, find_packages
 
 setup(name='tap-twilio',
-      version='0.0.3',
+      version='0.1.0',
       description='Singer.io tap for extracting data from the Twilio API',
       author='[email protected]',
       classifiers=['Programming Language :: Python :: 3 :: Only'],

diff --git a/tap_twilio/__init__.py b/tap_twilio/__init__.py
@@ -2,9 +2,7 @@
 
 import sys
 import json
-import argparse
 import singer
-from singer import metadata, utils
 from tap_twilio.client import TwilioClient
 from tap_twilio.discover import discover
 from tap_twilio.sync import sync
@@ -14,8 +12,7 @@
 REQUIRED_CONFIG_KEYS = [
     'account_sid',
     'auth_token',
-    'start_date',
-    'user_agent'
+    'start_date'
 ]
 
 def do_discover():

diff --git a/tap_twilio/schemas/conference_participants.json b/tap_twilio/schemas/conference_participants.json
@@ -22,6 +22,9 @@
     "end_conference_on_exit": {
       "type": ["null", "boolean"]
     },
+    "label": {
+      "type": ["null", "string"]
+    },
     "muted": {
       "type": ["null", "boolean"]
     },

diff --git a/tap_twilio/schemas/conferences.json b/tap_twilio/schemas/conferences.json
@@ -8,6 +8,9 @@
     "api_version": {
       "type": ["null", "string"]
     },
+    "call_sid_ending_conference": {
+      "type": ["null", "string"]
+    },
     "date_created": {
       "type": ["null", "string"],
       "format": "date-time"
@@ -22,6 +25,9 @@
     "sid": {
       "type": ["null", "string"]
     },
+    "reason_conference_ended": {
+      "type": ["null", "string"]
+    },
     "region": {
       "type": ["null", "string"]
     },

diff --git a/tap_twilio/schemas/incoming_phone_numbers.json b/tap_twilio/schemas/incoming_phone_numbers.json
@@ -103,6 +103,9 @@
     "emergency_address_sid": {
       "type": ["null", "string"]
     },
+    "emergency_address_status": {
+      "type": ["null", "string"]
+    },
     "address_sid": {
       "type": ["null", "string"]
     },

diff --git a/tap_twilio/schemas/messages.json b/tap_twilio/schemas/messages.json
@@ -48,6 +48,9 @@
       "type": ["null", "number"],
       "multipleOf": 1e-8
     },
+    "price_unit": {
+      "type": ["null", "string"]
+    },
     "sid": {
       "type": ["null", "string"]
     },

diff --git a/tap_twilio/streams.py b/tap_twilio/streams.py
@@ -25,6 +25,16 @@
         'params': {},
         'pagingation': 'root',
         'children': {
+            'account_balance': {
+                'api_url': 'https://api.twilio.com',
+                'api_version': '2010-04-01',
+                'path': 'Accounts/{ParentId}/Balance.json',
+                'data_key': 'account_balance',
+                'sub_resource_key': 'balance',
+                'key_properties': ['account_sid'],
+                'replication_method': 'FULL_TABLE',
+                'params': {}
+            },
             # pylint: disable=line-too-long
             # Reference: https://www.twilio.com/docs/usage/api/address#read-multiple-address-resources
             'addresses': {
@@ -146,6 +156,7 @@
                 'replication_method': 'INCREMENTAL',  # Fetch ALL, filter results
                 'replication_keys': ['date_updated'],
                 'params': {},
+                'parent': 'accounts',
                 'pagingation': 'root'
             },
             # pylint: disable=line-too-long
@@ -286,9 +297,8 @@
                 'data_key': 'usage_records',
                 'key_properties': ['account_sid', 'category', 'start_date'],
                 'replication_method': 'INCREMENTAL',  # Filter query
-                'replication_keys': ['end_date'],
-                'bookmark_query_field_from': 'start_date',  # Daily
-                'bookmark_query_field_to': 'end_date',
+                'replication_keys': ['start_date'],
+                'bookmark_query_field_from': 'StartDate',  # Daily
                 'params': {},
                 'pagingation': 'root'
             },
@@ -318,10 +328,11 @@
         'key_properties': ['sid'],
         'replication_method': 'INCREMENTAL',  # Filter query
         'replication_keys': ['date_updated'],
-        'bookmark_query_field_from': 'start_date',  # Bookmark
-        'bookmark_query_field_to': 'end_date',  # Current Date
+        'bookmark_query_field_from': 'StartDate',  # Bookmark
+        'bookmark_query_field_to': 'EndDate',
         'max_days_ago': 30,
         'params': {},
+        'pagination_key': 'next_page_url',
         'pagingation': 'meta'
     }
 }

diff --git a/tap_twilio/sync.py b/tap_twilio/sync.py
@@ -76,13 +76,16 @@ def process_records(catalog,  # pylint: disable=too-many-branches
         for record in records:
             # If child object, add parent_id to record
             if parent_id and parent:
-                record[parent + '_id'] = parent_id
+                if parent == 'accounts':
+                    record['account_sid'] = parent_id
+                else:
+                    record[parent + '_id'] = parent_id
 
             # Transform record for Singer.io
             with Transformer() as transformer:
                 try:
                     transformed_record = transformer.transform(
-                        record,
+                        dict(record),
                         schema,
                         stream_metadata)
                 except Exception as err:
@@ -173,7 +176,8 @@ def sync_endpoint(
         date_window_days=None,
         parent=None,
         parent_id=None,
-        account_sid=None):
+        account_sid=None,
+        required_streams=None):
     static_params = endpoint_config.get('params', {})
     bookmark_query_field_from = endpoint_config.get('bookmark_query_field_from')
     bookmark_query_field_to = endpoint_config.get('bookmark_query_field_to')
@@ -202,8 +206,9 @@ def sync_endpoint(
 
         params = static_params  # adds in endpoint specific, sort, filter params
 
-        if bookmark_query_field_from and bookmark_query_field_to:
+        if bookmark_query_field_from:
             params[bookmark_query_field_from] = strftime(start_window)[:10]  # truncate date
+        if bookmark_query_field_to:
             params[bookmark_query_field_to] = strftime(end_window)[:10]  # truncate date
 
         # pagination: loop thru all pages of data using next (if not None)
@@ -248,8 +253,10 @@ def sync_endpoint(
                 break  # No data results
 
             # Get pagination details
-            if data.get("next_page_uri"):
-                next_url = endpoint_config.get("api_url") + data["next_page_uri"]
+            # Next page url key in API response is different for alerts and remaining streams
+            next_url_key_in_api_response = endpoint_config.get('pagination_key', 'next_page_uri')
+            if data.get(next_url_key_in_api_response):
+                next_url = endpoint_config.get("api_url") + data[next_url_key_in_api_response]
             else:
                 next_url = None
 
@@ -301,8 +308,9 @@ def sync_endpoint(
             children = endpoint_config.get('children')
             if children:
                 for child_stream_name, child_endpoint_config in children.items():
-                    # will this work if only grandchildren are selected
-                    if child_stream_name in selected_streams:
+                    # Following check will make sure tap extracts the data for all the child streams
+                    # even if the parent isn't selected
+                    if child_stream_name in selected_streams or child_stream_name in required_streams:
                         LOGGER.info('START Syncing: {}'.format(child_stream_name))
                         write_schema(catalog, child_stream_name)
                         # For each parent record
@@ -333,7 +341,7 @@ def sync_endpoint(
                                         ParentId=parent_id)
                             elif child_stream_name == 'dependent_phone_numbers':
                                 child_path = child_endpoint_config.get('path').format(
-                                    ParentId=parent_id, AccountSid=config.get('account_sid'))
+                                    ParentId=parent_id, AccountSid=record.get('account_sid'))
                             else:
                                 child_path = record.get('_subresource_uris', {}).get(
                                     child_endpoint_config.get('sub_resource_key',
@@ -355,10 +363,11 @@ def sync_endpoint(
                                     selected_streams=selected_streams,
                                     # The child endpoint may be an endpoint that needs to window
                                     # so we'll re-pull from the config here (or pass in the default)
-                                    date_window_days=int(config.get('date_window_days', '30')),
+                                    date_window_days=int(config.get('date_window_days') or'30'),
                                     parent=child_endpoint_config.get('parent'),
                                     parent_id=parent_id,
-                                    account_sid=account_sid)
+                                    account_sid=account_sid,
+                                    required_streams=required_streams)
                             else:
                                 LOGGER.info(
                                     'No child stream {} for parent stream {} in subresource uris'
@@ -486,7 +495,8 @@ def sync(client, config, catalog, state):
                 endpoint_config=endpoint_config,
                 bookmark_field=bookmark_field,
                 selected_streams=selected_streams,
-                date_window_days=int(config.get('date_window_days', '30')))
+                date_window_days=int(config.get('date_window_days') or '30'),
+                required_streams=required_streams)
 
             update_currently_syncing(state, None)
             LOGGER.info('FINISHED Syncing: {}, total_records: {}'.format(

diff --git a/tests/__init__.py b/tests/__init__.py
-Original file line number
+Diff line change
@@ Expand Up / @@ -50,4 +50,4 @@ workflows: @@
                     - master
         jobs:
           - build:
-              context: circleci-user
+              context: circleci-user