Out of memory on a relatively small TypeScript project

[alecmev]

Hi! First things first, thanks for making Knip, I really like the mission!

I'm setting it up in our monorepo and have run into a few problems. This is the only showstopper, but I'll open a few more issues for the other ones, hope you don't mind.

The package foo I'm running it on is somewhat heavy on TypeScript (@typescript-eslint is noticeably slower on it too, compared to everything else in our monorepo), but it has never caused us trouble like this.

I'm running Knip from the root of the repo, like so:

yarn knip --workspace packages/foo

Every package has its own tsconfig.json, for foo it looks like so:

{
  "extends": [
    "@tsconfig/node20/tsconfig.json",
    "@tsconfig/strictest/tsconfig.json",
    "@tsconfig/esm/tsconfig.json"
  ],
  "include": [
    "src/**/*",
    "src/package.symlink.json"
  ],
  "compilerOptions": {
    "composite": true,
    "declaration": true,
    "declarationMap": true,
    "experimentalDecorators": true,
    "moduleResolution": "Node16",
    "resolveJsonModule": true,
    "sourceMap": true,
    "types": ["jest", "jest-image-snapshot", "jest-matcher-deep-close-to"],
    "jsx": "react-jsx",
    "lib": ["dom"],
    "outDir": "dist",
    "rootDir": "src",
    "tsBuildInfoFile": "dist/.tsbuildinfo"
  },
  "references": [
    { "path": "../bar" }
  ]
}

All packages in the repo have the same extends and the same compiler options above jsx. I've tried removing references, src/package.symlink.json, "lib": ["dom"], to no avail. The outcome looks like so:

# ...

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

# ...

I've tried ignoring all files except for index.ts, in which case Knip doesn't run out of memory, but is still extremely slow:

yarn knip --workspace packages/foo --debug --performance

# ...

[knip] Analyzing used resolved files [P1/1] (2)
[
  '/root/knip.ts',
  '/root/packages/foo/src/index.ts'
]

# ...

Name                        size  min       max       median    sum
--------------------------  ----  --------  --------  --------  --------
createProgram                  1  17709.86  17709.86  17709.86  17709.86
getTypeChecker                 1    329.04    329.04    329.04    329.04
glob                           6      0.01     52.21     18.69    133.29
load                           1     81.39     81.39     81.39     81.39
findManifestDependencies       1     18.65     18.65     18.65     18.65
getDependenciesFromScripts     2      0.01     13.50      6.76     13.51
firstGlob                      1      6.38      6.38      6.38      6.38
require                       14      0.11      0.57      0.23      3.52
dirGlob                        1      0.39      0.39      0.39      0.39

Total running time: 18.3s

For comparison, here's another similarly heavy package, with nothing ignored:

Name                        size  min     max     median  sum
--------------------------  ----  ------  ------  ------  ------
createProgram                  1  847.20  847.20  847.20  847.20
findReferences                13    2.31  723.48    5.99  812.95
getTypeChecker                 1  221.19  221.19  221.19  221.19
glob                           6    0.00   58.25   22.22  151.20
load                           1   83.69   83.69   83.69   83.69
findManifestDependencies       1   21.35   21.35   21.35   21.35
getDependenciesFromScripts    18    0.00   14.30    0.01   14.45
firstGlob                      1    9.39    9.39    9.39    9.39
require                       17    0.12    0.98    0.25    4.85
dirGlob                        1    0.41    0.41    0.41    0.41

Total running time: 2.3s

If I un-ignore more than 1 file beyond index.ts then it runs out of memory again.

It appears to me that Knip is using the right tsconfig.json, and this probably isn't Knip's fault at all, but I'd like to have a little bit more transparency into how it interacts with TypeScripts' language server. Thinking of setting up some console.logs around createProgram, to see the exact compiler settings, but reporting here in case you have some pointers.

Latest everything.

Thanks!

[webpro]

Knip is trying to be smart and win performance by combining TS configs of packages that allow so, since ts.createProgram is slow. To be honest, this went quite well so far: https://github.com/webpro/knip/blob/main/src/PrincipalFactory.ts#L28-L31

Unfortunately in your situation it means the opposite.

One potential solution is to offer a flag to disable the optimization, this I will do shortly. And maybe have a way to detect or set a maximum number of TS projects that can be "merged" this way.

Any chance you could share the project so I can try and test things?

[alecmev]

Let me try creating a repro first. I don't think the optimization is at fault here, as the slowdown/crash happens even when invoking Knip in just that workspace.

[webpro]

Ah yes I read that too fast then. Still I think the option is good to have, to (1) spread cpu/memory usage differently, and (2) offer a workaround for potential issues arising from merging the TS configs (e.g. from basePath and paths stuff).

[alecmev]

Reproduced: https://github.com/alecmev/knip-repro-183 It is an ancient version of Babylon.js, but yarn tsc works fine. yarn knip is enough to trigger the crash.

[webpro]

Thanks for the repro! Here to say I definitely will dig into this more when I have time.

First exploration showed that the dependency has like 180 nested dts imports and ts.createSourceFile (createProgram) and ts.ScriptSnapshot.fromString (findReferences) read and parse all of them (once). Perhaps tsc has some optimizations there that Knip doesn't have. It didn't crash for me, but definitely shouldn't take so long indeed.

[alecmev]

Thank you! We might upgrade to the new Babylon.js soon, alleviating this, but this is still worth looking into. Also, if it doesn't crash on your system, then try creating more copies of bar.ts and re-exporting them in index.ts 😄

[alecmev]

Updated to Babylon.js 6, and fixed all errors reported with skipLibCheck: false, and it works like a charm, closing 😉

[webpro]

Yeah, sorry, didn't get to it. Also not hearing anyone else complain about performance, so there was not much incentive to look into a single case. Glad it resolved itself :)

[nicooprat]

Actually, I subscribed here because we also have memory usage issues in Github Actions.

We don't use directly babylon, but as a nested dep it's already version 6.18.0.

We temporarily fixed it by prefixing our script with NODE_OPTIONS=--max-old-space-size=5120.

I must say our project is not as small as the OP.

Didn't have time to look into skipLibCheck yet (which is true in our tsconfig.json).

Just sayin', because I read "not hearing anyone else complain about performance" ;)

[alecmev]

Just in case, clean skipLibCheck: false could be a red herring! Though it is a good idea, even if it doesn't help. In our case it uncovered the fact that Babylon.js isn't compatible with moduleResolution: Node16, which was resulting in all sorts of weird type-checking problems (e.g., classes missing the members of the classes they're extending), so for now we're using Bundler (which requires a soon-to-be-released patch to function).

[webpro]

Knip v4 is about to mitigate some of these issues: https://knip.dev/blog/slim-down-to-speed-up (you can install and try v4 canary today). Recently added --isolate-workspaces (in v3) which may also help in certain edge cases.