Build out a cli tool that does our own parsing of javascript files, d…

…ramatically speeding things up! (#20) Summary: Using jest, it was taking 25 seconds to run everything, and using this custom parser, it only takes 6! Much more reasonable. And we could also likely improve things further by only processing operations in files that have been edited. Issue: https://khanacademy.atlassian.net/browse/FEI-4426 Test plan: `yarn jest`, also ran this on all of webapp and it produced the same types as the jest version! And great coverage! ``` ---------------------------------|---------|----------|---------|--------- File | % Stmts | % Branch | % Funcs | % Lines ---------------------------------|---------|----------|---------|--------- src/parser | 97.12 | 80.55 | 100 | 97.09 parse.js | 97.65 | 78.22 | 100 | 97.61 resolve.js | 95.65 | 95 | 100 | 95.65 ---------------------------------|---------|----------|---------|--------- ``` Author: jaredly Reviewers: jaredly, benchristel, kevinbarabash, jeremywiebe Required Reviewers: Approved By: benchristel, jeremywiebe Checks: ✅ Lint & Test (ubuntu-latest, 16.x) Pull Request URL: #20
Khan · Apr 6, 2022 · b08ed1b · b08ed1b
1 parent 8cdcdc2
commit b08ed1b
Show file tree

Hide file tree

Showing 13 changed files with 1,216 additions and 217 deletions.
diff --git a/.changeset/popular-cycles-judge.md b/.changeset/popular-cycles-judge.md
@@ -0,0 +1,5 @@
+---
+'@khanacademy/graphql-flow': minor
+---
+
+Build out a cli tool that does our own parsing of javascript files, dramatically speeding things up!
diff --git a/.gitignore b/.gitignore
@@ -1,2 +1,3 @@
 node_modules
+coverage
 dist
diff --git a/Readme.md b/Readme.md
@@ -2,64 +2,67 @@
 
 This is a tool for generating flow types from graphql queries in javascript frontends.
 
-The core of this tool is the `documentToFlowTypes` function, which takes a `DocumentNode` (such as is returned by the `graphql-tag` package) as well as your backend's graphql schema (see below for instructions on how to produce this), and produces a list of stringified flow types, one for each query and mutation defined in that graphql block.
+## Using as a CLI tool
 
-It looks something like this:
+Write a config file, with the following options:
 
-```js
-import gql from 'graphql-tag';
-import {documentToFlowTypes, schemaFromIntrospectionData} from 'graphql-flow';
-import myIntrospectionData from './server-introspection-response.json';
-
-const schema = schemaFromIntrospectionData(myIntrospectionData);
-
-const MyQuery = gql`
-    query SomeQuery {
-        human(id: "Han Solo") {
-            id
-            name
-            homePlanet
-            friends {
-                name
-            }
+```json
+{
+    // Response to the "introspection query" (see below). This path is resolved relative to the config file location.
+    "schemaFilePath": "../some/schema-file.json",
+    // List of regexes
+    "excludes": ["\\bsome-thing", "_test.jsx?$"],
+    // Options for type generation (see below)
+    "options": {
+        "scalars": {
+            "JSONString": "string"
         }
     }
-`;
-
-console.log(documentToFlowTypes(MyQuery, schema))
-/*
-export type SomeQueryResponseType = {|
-    human: ?{|
-        id: string,
-        name: ?string,
-        homePlanet: ?string,
-        friends: ?$ReadOnlyArray<?{|
-            name: ?string
-        |}>,
-    |}
-|};
-*/
+}
 ```
 
-If you already have a setup whereby you collect all of your graphql literals, that may be all you need!
+Then run from the CLI, like so:
 
-Otherwise, we provide a way to hook into jest to automatically collect your queries and generate the types.
+```bash
+$ graphql-flow path/to/config.json
+```
 
-## Options for `documentToFlowTypes`
+Files will be discovered relative to the current working directory.
 
-```js
-{
-    // Use nullable types where the graphql type is nullable. Included for legacy compatability,
-    // will probably remove once the mobile repo no longer needs it.
+To specify what file should be checked, pass them in as subsequent cli arguments.
+
+## Options (for the cli 'options' config item, or when running from jest):
+
+```ts
+type Options = {
+    // These are from the `documentToFlowTypes` options object above
     strictNullability: boolean = true,
-    // Output `$ReadOnlyArray<>` instead of `Array<>`, for stricter flow typing. On by default.
     readOnlyArray: boolean = true,
-    // A mapping of custom scalar names to the underlying json representation.
     scalars: {[key: string]: 'string' | 'boolean' | 'number'}
+
+    // Specify an opt-in pragma that must be present in a graphql string source
+    // in order for it to be picked up and processed
+    // e.g. set this to `"# @autogen\n"` to only generate types for queries that
+    // have the comment `# @autogen` in them.
+    pragma?: string,
+    // Specify a pragma that will turn off `strictNullability` for that
+    // source file. e.g. `"# @autogen-loose\n"`.
+    loosePragma?: string,
+    // If neither pragma nor loosePragma are specified, all graphql documents
+    // that contain a query or mutation will be processed.
+
+    // Any graphql operations containing ignorePragma will be skipped
+    ignorePragma?: string,
 }
 ```
 
-## Using jest to do the heavy lifting:
+## Using from jest
+
+You can also use jest to do the heavy lifting, running all of your code and collecting queries
+by mocking out the `graphql-tag` function itself. This requires that all graphql operations are
+defined at the top level (no queries defined in functions or components, for example), but does
+support complicated things like returning a fragment from a function (which is probably
+not a great idea code-style-wise anyway).
 
 ### jest-setup.js
 
@@ -74,6 +77,7 @@ if (process.env.GRAPHQL_FLOW) {
 
         return jest.requireActual('../tools/graphql-flow/jest-mock-graphql-tag.js')(
             introspectionData,
+            // See "Options" type above
             {
                 pragma: '# @autogen\n',
                 loosePragma: '# @autogen-loose\n',
@@ -126,31 +130,6 @@ And then `yarn generate-types` or `npm run generate-types` gets your types gener
 
 🚀
 
-### Options for the `jest-mock-graphql-tag.js` helper:
-
-```js
-{
-    // These are from the `documentToFlowTypes` options object above
-    strictNullability: boolean = true,
-    readOnlyArray: boolean = true,
-    scalars: {[key: string]: 'string' | 'boolean' | 'number'}
-
-    // Specify an opt-in pragma that must be present in a graphql string source
-    // in order for it to be picked up and processed
-    // e.g. set this to `"# @autogen\n"` to only generate types for queries that
-    // have the comment `# @autogen` in them.
-    pragma?: string,
-    // Specify a pragma that will turn off `strictNullability` for that
-    // source file. e.g. `"# @autogen-loose\n"`.
-    loosePragma?: string,
-    // If neither pragma nor loosePragma are specified, all graphql documents
-    // that contain a query or mutation will be processed.
-
-    // Any graphql operations containing ignorePragma will be skipped
-    ignorePragma?: string,
-}
-```
-
 ## Introspecting your backend's graphql schema
 Here's how to get your backend's schema in the way that this tool expects, using the builtin 'graphql introspection query':
 

diff --git a/package.json b/package.json
@@ -1,10 +1,13 @@
 {
     "name": "@khanacademy/graphql-flow",
     "version": "0.0.2",
+    "bin": {
+        "graphql-flow": "./dist/cli/run.js"
+    },
     "scripts": {
         "test": "jest",
         "publish:ci": "yarn run build && yarn run copy-flow && changeset publish",
-        "build": "babel src --out-dir dist --ignore 'src/**/*.spec.js','src/**/*.test.js'",
+        "build": "babel src --out-dir dist --source-maps --ignore 'src/**/*.spec.js','src/**/*.test.js'",
         "copy-flow": "node ./build-copy-source.js"
     },
     "main": "dist/index.js",
@@ -14,9 +17,9 @@
         "@babel/polyfill": "^7.0.0",
         "@babel/preset-env": "^7.16.11",
         "@babel/preset-flow": "^7.16.7",
-        "babel-jest": "23.4.2",
         "@changesets/cli": "^2.21.1",
         "@khanacademy/eslint-config": "^0.1.0",
+        "babel-jest": "23.4.2",
         "eslint": "8.7.0",
         "eslint-config-prettier": "7.0.0",
         "eslint-plugin-flowtype": "^8.0.3",
@@ -25,13 +28,14 @@
         "eslint-plugin-prettier": "^4.0.0",
         "eslint-plugin-react": "^7.29.2",
         "eslint-plugin-react-hooks": "^4.3.0",
-        "graphql-tag": "2.10.1",
         "flow-bin": "^0.172.0",
+        "graphql-tag": "2.10.1",
         "jest": "^27.5.1"
     },
     "dependencies": {
         "@babel/core": "^7.6.2",
         "@babel/generator": "^7.17.3",
+        "@babel/traverse": "^7.17.3",
         "@babel/types": "^7.17.0",
         "apollo-utilities": "^1.3.4",
         "graphql": "14.2.1",

diff --git a/src/__test__/jest-mock-graphql-tag.test.js → src/__test__/processPragmas.test.js b/src/__test__/jest-mock-graphql-tag.test.js → src/__test__/processPragmas.test.js
@@ -1,5 +1,5 @@
 // @flow
-import {processPragmas} from '../jest-mock-graphql-tag';
+import {processPragmas} from '../generateTypeFiles';
 
 const pragma = '# @autogen\n';
 const loosePragma = '# @autogen-loose\n';

diff --git a/src/cli/config.js b/src/cli/config.js
@@ -0,0 +1,64 @@
+// @flow
+import type {ExternalOptions} from '../generateTypeFiles';
+
+import fs from 'fs';
+import path from 'path';
+
+export type CliConfig = {
+    excludes: Array<RegExp>,
+    schemaFilePath: string,
+    options: ExternalOptions,
+};
+
+/**
+ * This is the json-compatible form of the config
+ * object.
+ */
+type JSONConfig = {
+    excludes?: Array<string>,
+    schemaFilePath: string,
+    options?: ExternalOptions,
+};
+
+export const loadConfigFile = (configFile: string): CliConfig => {
+    // eslint-disable-next-line flowtype-errors/uncovered
+    const data: JSONConfig = JSON.parse(fs.readFileSync(configFile, 'utf8'));
+    const toplevelKeys = ['excludes', 'schemaFilePath', 'options'];
+    Object.keys(data).forEach((k) => {
+        if (!toplevelKeys.includes(k)) {
+            throw new Error(
+                `Invalid attribute in config file ${configFile}: ${k}. Allowed attributes: ${toplevelKeys.join(
+                    ', ',
+                )}`,
+            );
+        }
+    });
+    if (data.options) {
+        const externalOptionsKeys = [
+            'pragma',
+            'loosePragma',
+            'ignorePragma',
+            'scalars',
+            'strictNullability',
+            'regenerateCommand',
+            'readOnlyArray',
+        ];
+        Object.keys(data.options).forEach((k) => {
+            if (!externalOptionsKeys.includes(k)) {
+                throw new Error(
+                    `Invalid option in config file ${configFile}: ${k}. Allowed options: ${externalOptionsKeys.join(
+                        ', ',
+                    )}`,
+                );
+            }
+        });
+    }
+    return {
+        options: data.options ?? {},
+        excludes: data.excludes?.map((string) => new RegExp(string)) ?? [],
+        schemaFilePath: path.join(
+            path.dirname(configFile),
+            data.schemaFilePath,
+        ),
+    };
+};