Testing for Memory Leaks

One of the more tricky kinds of bugs to track down in Python programs (including Crystal) is a memory leak. Crystal allocates and deallocates a huge number of objects over its lifetime while it is downloading new URL resources to a project.

Here, I'll outline a general strategy of:

• Get an interactive Python interpreter inside the Python process that is running Crystal
• Start Crystal with a large project and do a complex operation (like downloading a large group)
• In that interpreter, use Heapy (guppy3) to profile memory use

Get an interactive Python interpreter running inside the Crystal process

Method 0: Launch Crystal with interpreter using --shell

Just use Crystal's built-in shell:

poetry run python3 src/main.py --shell

Method 1: Alter source of Python program to add interpreter

In src/main.py just before # Run GUI add the lines:

import code, threading; threading.Thread(target=code.interact).start()

Then when you run Crystal from source, using python src/main.py, you'll immediately get a Python interpreter on the command-line in addition to the regular GUI.

Method 2: Attach to running Python process with pyrasite

Use pyrasite to attach to an existing process. Under the covers it uses the gdb tool to attach to a running Python process - and therefore has the same limitations as that approach.

pyrasite works well on Linux (including Docker containers) and older versions of macOS that have fewer application sandboxing features. I doubt it works on Windows since it uses gdb.

Inside a Docker container, run with docker run --cap-add=SYS_PTRACE ...:

$ PYTHON_CRYSTAL_PID=...
$ pip3 install pyrasite ; apt-get update ; apt-get install -y gdb
$ pyrasite-shell $PYTHON_CRYSTAL_PID

On macOS, you'll need to additionally codesign both gdb and the python binary for the version of Crystal you're trying to attach to. And for Python 3.8 it appears you need to use a Python process with debugging symbols included, which I haven't figured out how to make work.

In general, when debugging pyrasite, try to run:

$ PYTHON_CRYSTAL_PID=...
$ gdb -p $PYTHON_CRYSTAL_PID  # can fail if ptrace unavailable, binary is unsigned (if macOS), or other reasons
(gbd) call PyGILState_Ensure()  # can fail if Python not compiled with debugging symbols
(gbd) call PyRun_SimpleString("print('success')")
(gbd) call PyGILState_Release($1)
(gbd) quit

Method 3: Alter source of Python program with custom signal handler that opens an interpreter

https://stackoverflow.com/a/133384/604063

Start Crystal and do something complex

• Run Crystal from source
  • $ python3 src/main.py
• Open a large project, like "OtakuWorld + The Big KiSS Page + Jenniverse.crystalproj"
• Start downloading a large group like "Jenniverse Forum Topic" (26,394 members)
• While the download is in progress, use the interactive Python interpreter to profile the running process

Profile memory use of Crystal with Heapy (guppy3)

You will need to have Heapy installed first. Crystal 1.6.0 and later includes it by default.

$ pip3 install guppy3

In the interactive Python interpreter, import Heapy once:

>>> from guppy import hpy; h = hpy()

Then whenever you want to take a memory sample, use:

>>> import gc; gc.collect(); heap = h.heap(); heap; _.more
3135053
Partition of a set of 4679418 objects. Total size = 527,707,524 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 294638   6 71062223  13  71062223  13 str
     1 105108   2 65587392  12 136649615  26 collections.deque
     2 735874  16 51500536  10 188150151  36 list
     3 528338  11 38040336   7 226190487  43 types.BuiltinMethodType
     4 105108   2 37838240   7 264028727  50 dict of threading.Condition
     5 317136   7 32982144   6 297010871  56 dict of crystal.model._WeakTaskRef
     6 222763   5 30295768   6 327306639  62 wx._core.TreeItemId
     7 144139   3 20756016   4 348062655  66 dict of crystal.ui.tree.NodeView
     8 317136   7 15222528   3 363285183  69 crystal.model._WeakTaskRef
     9 105712   2 15222528   3 378507711  72 dict of crystal.model.Resource
<482 more rows. Type e.g. '_.more' to view.>
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
    10 105105   2 15135120   3 393642831  75 dict of concurrent.futures._base.Future
    11 222763   5 14256832   3 407899663  77 crystal.ui.tree.NodeViewPeer
    12  78624   2 11321856   2 419221519  79 dict of crystal.ui.tree2.NodeView
    13      2   0 10732560   2 429954079  81 collections.OrderedDict
    14  72033   2  7491432   1 437445511  83 dict of crystal.browser.entitytree._LoadingNode
    15  71915   2  7479160   1 444924671  84 dict of crystal.browser.entitytree.NormalResourceNode
    16 144139   3  6918672   1 451843343  86 crystal.ui.tree.NodeView
    17  65790   1  6842160   1 458685503  87 dict of crystal.browser.tasktree.TaskTreeNode
    18  26481   1  6143592   1 464829095  88 dict of crystal.task.DownloadResourceBodyTask
    19  26481   1  6143592   1 470972687  89 dict of crystal.task.DownloadResourceTask
<472 more rows. Type e.g. '_.more' to view.>

If any object types seem suspicious, examine the shortest path from one of the objects to the GC root:

>>> heap[9].byid[0].shpaths  # Resource (not actually suspicious)
 0: hpy().Root.??.f_back.f_back.f_back.f_back.f_locals['task'].__dict__['_abstract_resource'].__dict__['project'].__dict__['_resources']['http://otakuworld.com/'].__dict__
 1: hpy().Root.??.f_back.f_back.f_back.f_back.f_locals['task'].__dict__['_abstract_resource'].__dict__['project'].__dict__['_root_resources'].keys()[0].__dict__
 2: hpy().Root.??.f_back.f_back.f_back.f_back.f_locals['task'].__dict__['_resource'].__dict__['project'].__dict__['_resources']['http://otakuworld.com/'].__dict__
 3: hpy().Root.??.f_back.f_back.f_back.f_back.f_locals['task'].__dict__['_resource'].__dict__['project'].__dict__['_root_resources'].keys()[0].__dict__

Here an executing Python stack frame (via chains of f_back) has a local variable task that contains a chain of references eventually to a Project object which contains all Resource objects (including this particular Resource object).

You can also profile memory growth specifically by setting a checkpoint. Then subsequent memory samples will only show new objects since the checkpoint:

>>> h.setref()  # create checkpoint

Heapy also documents a h.heapu() call that can hypothetically see leaked native C/C++ objects (such as from wxPython), but it doesn't seem to give readable output in guppy3. Maybe the original guppy module (which doesn't support Python 3.x) has a working version...