Add support for Twitter API v2 Full-Archive Search (#18)

As-is, Harassment Manager leverages the Enterprise Full-Archive Search API to fetch tweets directed at the logged-in user. This change enables developers to use either Enterprise or Twitter API v2 Full-Archive Search, which is the latest version of the Twitter API. To implement this, we add the relevant request logic to `twitter.middleware.ts`. We "pack" the v2 response format into the Enterprise response format to enable developers to switch between Enterprise and v2 forwards and backwards, without breaking usage for existing users of the application. Additional changes include: - Removing unused types and defining new ones for the v2 format - Changing `fromDate` and `toDate` to `startDateTimeMs` and `endDateTimeMs` and using formatting functions to accommodate differences in the timestamps expected by Enterprise vs. v2 - Modifying `server_config_template.json` with a `bearerToken` field, which is necessary for using v2 Full-Archive Search - Updating the AppEngine runtime to `nodejs12` to support some new operators, like `flatMap` - Updating documentation We deployed and tested an instance of Harassment Manager with each API and the application's behavior is largely identical between both versions, with a couple minor differences: 1. The APIs return slightly different sets of tweets because of differences in the granularity of the timestamp format expected by each API. This difference is usually no more than an additional 1-3 tweets in the v2 instance. 2. The tweets are displayed in slightly different orders when sorted by "Priority". This is due to small differences in how we parse the tweet text, which causes some variation in the Perspective API scores for the text. We opened issue #19 to investigate. We also noticed the Enterprise instance does not render some images that the v2 instance does. This seems more like an implementation issue on our end, rather than an API difference. We opened issue #17 to investigate.
conversationai · Dec 1, 2022 · 6c9a501 · 6c9a501
1 parent 9f8c847
commit 6c9a501
Show file tree

Hide file tree

Showing 13 changed files with 420 additions and 104 deletions.
diff --git a/.gcloudignore b/.gcloudignore
@@ -14,4 +14,7 @@
 .gitignore
 
 # Node.js dependencies:
-node_modules/
+node_modules/
+
+# Miscellaneous
+.angular/cache
diff --git a/app.yaml b/app.yaml
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-runtime: nodejs10
+runtime: nodejs12
 service: default
 instance_class: F4_1G
 
@@ -25,7 +25,7 @@ automatic_scaling:
 
 # Required for min_idle_instances!
 inbound_services:
-- warmup
+  - warmup
 
 env_variables:
   NODE_ENV: production
diff --git a/docs/1_setup.md b/docs/1_setup.md
@@ -18,40 +18,61 @@ of developers.
 
 ## 1. Get access to Twitter APIs
 
-**NOTE: The full suite of Twitter APIs the app uses require additional access
-beyond the default Twitter API [Essential access
-level](https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api).
-The [Enterprise
-search](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview)
-additionally requires an [enterprise
-account](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview).**
-
-**NOTE: We plan to migrate the Enterprise Full-Archive Search API to the v2 Search Tweets
-in the future. We will update this documentation accordingly.**
-
 The app makes use of several Twitter APIs, including:
 
-- [The Enterprise Full-Archive Search
-  API](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview) to fetch
-  tweets directed at the logged in user
-- The v2 [blocks](https://developer.twitter.com/en/docs/twitter-api/users/blocks/introduction)
+- The [Enterprise Full-Archive Search
+  API](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview)
+  **or** the [v2 Full-Archive Search
+  endpoint](https://developer.twitter.com/en/docs/twitter-api/tweets/search/quick-start/full-archive-search)
+  to fetch tweets directed at the logged in user
+- The v2
+  [blocks](https://developer.twitter.com/en/docs/twitter-api/users/blocks/introduction)
   endpoint to block users on behalf of the authenticated user
-- The v2 [mutes](https://developer.twitter.com/en/docs/twitter-api/users/mutes/introduction)
+- The v2
+  [mutes](https://developer.twitter.com/en/docs/twitter-api/users/mutes/introduction)
   endpoint to mute users on behalf of the authenticated user
 - The v2 [hide
   replies](https://developer.twitter.com/en/docs/twitter-api/tweets/hide-replies/introduction)
   endpoint to hide replies on behalf of the authenticated user
 
 To support all this functionality, you'll need to [get access to the Twitter
-API](https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api)
-and the Enterprise Full-Archive Search API.
+API](https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api).
 
-Once granted, take note of the:
+**NOTE: The full suite of Twitter APIs the app uses require additional access
+beyond the default Twitter API [Essential access
+level](https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api).
+The [Enterprise
+search](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview)
+API requires an [enterprise
+account](https://developer.twitter.com/en/docs/twitter-api/enterprise/search-api/overview),
+while v2 Full-Archive Search requires [academic
+access](https://developer.twitter.com/en/products/twitter-api/academic-research).**
 
-- Account name, app key, and app secret for your Twitter API developer account
-- Username and password for your Enterpise Full-Archive Search API account
+Once granted access, take note of the:
 
-You'll need both sets of credentials later on.
+- Account name, app key, and app secret for your Twitter API developer account
+- If using the Enterprise Full-Archive Search API, the Username and password for
+  your enterprise account
+- If using the v2 Full-Archive Search endpoint, the Twitter bearer token for
+  your app
+
+You'll need these credentials later on.
+
+### Enterprise Full-Archive Search vs. v2 Full-Archive Search
+
+The tool is implemented in a way that either API can be used. While both APIs
+offer similar functionality, there are key differences in rate limits. We refer
+users to [Twitter's
+comparison](https://developer.twitter.com/en/docs/twitter-api/tweets/search/migrate)
+for more details. You may also see minute differences in:
+
+- Which tweets are fetched. This is due to differences in granularity for
+  timestamp format each API supports (YYYYMMDD for Enterprise and
+  YYYY-MM-DDTHH:mm:ssZ for v2).
+- The order the tweets are displayed when sorted by "Priority". This is due to
+  small differences in how we parse out the tweet text, which causes some
+  variation in the Perspective API scores for the text. See issue #19 for more
+  details.
 
 ## 2. Create a Google Cloud Platform (GCP) project
 

diff --git a/docs/2_development.md b/docs/2_development.md
@@ -145,16 +145,16 @@ The required fields are:
   be the server-side key created in [setup](1_setup.md) in GCP
   [Credentials](https://console.cloud.google.com/apis/credentials)
 - `cloudProjectId`: Your Google Cloud project ID
+- `twitterApiCredentials`: Your credentials for the Twitter APIs. For Enterprise
+  Full-Archive search, Twitter will provide you with the credentials. All other
+  API credentials should be available on the Twitter [Developer
+  Portal](https://developer.twitter.com/portal) under "Keys and Tokens" for your
+  app and project.
 
-The optional fields are:
+All together, your config should look something like one of the two configs
+below, with the relevant credentials and key values replaced.
 
-- `twitterApiCredentials`: Your credentials for the Twitter APIs. The server
-  expect this field to be an object with `accountName`, `username`, and
-  `password` fields for the Enterprise Search API and `appKey` and `appToken`
-  for the Standard API.
-
-All together, your config should look something like the config below, with the
-relevant credentials and key values replaced.
+### If using the Enteprise Full-Archive Search API:
 
 ```json
 {
@@ -166,8 +166,24 @@ relevant credentials and key values replaced.
     "accountName": "{TWITTER_API_ACCOUNT_NAME}",
     "username": "{TWITTER_API_USERNAME}",
     "password": "{TWITTER_API_PASSWORD}",
-    "appKey": "{APP_KEY}",
-    "appToken": "{APP_TOKEN}"
+    "appKey": "{TWITTER_APP_KEY}",
+    "appToken": "{TWITTER_APP_TOKEN}"
+  }
+}
+```
+
+### If using the v2 Full-Archive Search endpoint:
+
+```json
+{
+  "port": "3000",
+  "staticPath": "dist/harassment-manager",
+  "googleCloudApiKey": "{YOUR_GOOGLE_CLOUD_API_KEY}",
+  "cloudProjectId": "{YOUR_GOOGLE_CLOUD_PROJECTID}",
+  "twitterApiCredentials": {
+    "appKey": "{TWITTER_APP_KEY}",
+    "appToken": "{TWITTER_APP_TOKEN}",
+    "bearerToken": "{TWITTER_APP_BEARER_TOKEN}"
   }
 }
 ```
@@ -226,3 +242,7 @@ We maintain a [CircleCI](https://circleci.com/) configuration in
 is pushed to this GitHub repository. You can choose to use the same
 configuration for your own CircleCI setup if you'd like or remove the
 configuration in favor of another CI solution or none at all.
+
+```
+
+```
diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
@@ -105,7 +105,7 @@
     "@types/jasmine": "^3.10.3",
     "@types/jasminewd2": "~2.0.10",
     "@types/jspdf": "^1.3.3",
-    "@types/node": "^17.0.21",
+    "@types/node": "17.0.10",
     "@types/supertest": "^2.0.11",
     "@typescript-eslint/eslint-plugin": "5.14.0",
     "@typescript-eslint/parser": "5.14.0",

diff --git a/src/app/social-media-item.service.ts b/src/app/social-media-item.service.ts
@@ -126,11 +126,11 @@ export class SocialMediaItemService {
   ): Observable<Tweet[]> {
     return from(
       this.twitterApiService.getTweets({
-        fromDate: formatTimestamp(startDateTimeMs),
-        // Subtract 1 minute from the end time because the Twitter API
-        // sometimes returns an error if we request data for the most recent
-        // minute of time.
-        toDate: formatTimestamp(endDateTimeMs - 60000),
+        startDateTimeMs,
+        // Subtract 1 minute from the end time because the Twitter API sometimes
+        // returns an error if we request data for the most recent minute of
+        // time.
+        endDateTimeMs: endDateTimeMs - 60000,
       })
     ).pipe(map((response: GetTweetsResponse) => response.tweets));
   }
@@ -166,22 +166,3 @@ export class SocialMediaItemService {
       .pipe(map(scores => ({ item, scores })));
   }
 }
-
-// Format a millisecond-based timestamp into a date format suitable for the
-// Twitter API, as defined in:
-// https://developer.twitter.com/en/docs/tweets/search/api-reference/enterprise-search
-function formatTimestamp(ms: number): string {
-  const date = new Date(ms);
-  const MM = date.getUTCMonth() + 1; // getMonth() is zero-based
-  const dd = date.getUTCDate();
-  const hh = date.getUTCHours();
-  const mm = date.getUTCMinutes();
-
-  return (
-    `${date.getFullYear()}` +
-    `${(MM > 9 ? '' : '0') + MM}` +
-    `${(dd > 9 ? '' : '0') + dd}` +
-    `${(hh > 9 ? '' : '0') + hh}` +
-    `${(mm > 9 ? '' : '0') + mm}`
-  );
-}
diff --git a/src/app/test_constants.ts b/src/app/test_constants.ts
@@ -37,7 +37,6 @@ export const TWITTER_ENTRIES: Array<ScoredItem<Tweet>> = [
         user_mentions: [],
       },
       extended_tweet: {
-        display_text_range: [0, 279],
         entities: {
           hashtags: [
             {
@@ -110,7 +109,6 @@ export const TWITTER_ENTRIES: Array<ScoredItem<Tweet>> = [
         user_mentions: [],
       },
       extended_tweet: {
-        display_text_range: [0, 213],
         entities: {
           hashtags: [
             {
@@ -189,7 +187,6 @@ export const TWITTER_ENTRIES: Array<ScoredItem<Tweet>> = [
         ],
       },
       extended_tweet: {
-        display_text_range: [0, 203],
         entities: {
           hashtags: [],
           symbols: [],
@@ -274,7 +271,6 @@ export const TWITTER_ENTRIES: Array<ScoredItem<Tweet>> = [
         user_mentions: [],
       },
       extended_tweet: {
-        display_text_range: [0, 274],
         entities: {
           hashtags: [],
           symbols: [],

diff --git a/src/app/tweet-image/tweet-image.component.html b/src/app/tweet-image/tweet-image.component.html
@@ -16,17 +16,17 @@
 
 <ng-container *ngIf="tweet && thumbnail">
   <span class="image-thumbnail"
-        *ngIf="tweet.extended_entities && tweet.extended_entities.media">
+        *ngIf="tweet.extended_entities?.media?.length">
     <img [class.blur]="shouldBlur"
          [src]="tweet.extended_entities.media[0].media_url" [alt]="shouldBlur ? 'Blurred image from tweet.' : 'Image from tweet.'">
   </span>
 </ng-container>
 <ng-container *ngIf="tweet && !thumbnail">
   <div class="full-image"
        [class.collapsed]="collapsed"
-       *ngIf="tweet.extended_entities && tweet.extended_entities.media">
+       *ngIf="tweet.extended_entities?.media?.length">
     <img *ngFor="let media of tweet.extended_entities.media"
-        [class.blur]="shouldBlur" 
+        [class.blur]="shouldBlur"
         [src]="media.media_url" [alt]="shouldBlur ? 'Blurred image from tweet.' : 'Image from tweet.'">
   </div>
 </ng-container>