Memory leak: intopt and caller's frame not garbage collected (when ValueError raised)

[timdiels]

When running the reproduce script below (which causes intopt to raise ValueError), traceback, frame (codename: ecyglpki.Problem.intopt (ecyglpki.c:23524)), frame (codename: _leaky) objects are leaked. (The memory leaked is actually far greater than what the output reports below, it probably does not include C allocated memory).

Examining the referrers tree, frame(_leaky) is reffered to by frame(intopt) and traceback(1). frame(intopt) is reffered to by traceback(2); and traceback(2) is referred to by traceback(1) forming a cycle. Though, Python's GC supposedly handles cycles. If you run the reproduce script with iterations set to 5, you should get a Tk GUI allowing you to browse the referrer tree to the very_leaky frame.

Output (from reproduce script below). First object listing is before running intopt (multiple times). Second is directly after it. Last listing is after running gc.collect:

                               types |   # objects |   total size
==================================== | =========== | ============
                        <class 'dict |        2937 |      2.30 MB
                        <class 'type |         864 |    858.94 KB
                         <class 'set |         542 |    185.56 KB
                       <class 'tuple |        2432 |    165.21 KB
                     <class 'weakref |        1632 |    127.50 KB
          <class 'wrapper_descriptor |        1548 |    120.94 KB
                        <class 'list |         739 |    111.91 KB
           <class 'method_descriptor |        1286 |     90.42 KB
  <class 'builtin_function_or_method |        1173 |     82.48 KB
           <class 'getset_descriptor |        1131 |     79.52 KB
     <class 'collections.OrderedDict |          69 |     75.62 KB
                 <class 'abc.ABCMeta |          74 |     72.48 KB
                 function (__init__) |         435 |     57.77 KB
                   <class 'frozenset |          62 |     37.81 KB
           <class 'member_descriptor |         432 |     30.38 KB
                                                         types |   # objects |   total size
============================================================== | =========== | ============
                                      frame (codename: _leaky) |       50000 |     24.80 MB
  frame (codename: ecyglpki.Problem.intopt (ecyglpki.c:23524)) |       50000 |     19.07 MB
                                             <class 'traceback |      100000 |      6.10 MB
                                                  <class 'dict |        2856 |      2.21 MB
                                                  <class 'type |         864 |    858.94 KB
                                                  <class 'list |        5460 |    554.85 KB
                                                   <class 'set |         542 |    185.56 KB
                                                 <class 'tuple |        2348 |    155.85 KB
                                               <class 'weakref |        1632 |    127.50 KB
                                    <class 'wrapper_descriptor |        1548 |    120.94 KB
                                     <class 'method_descriptor |        1286 |     90.42 KB
                            <class 'builtin_function_or_method |        1173 |     82.48 KB
                                     <class 'getset_descriptor |        1131 |     79.52 KB
                               <class 'collections.OrderedDict |          69 |     75.62 KB
                                           <class 'abc.ABCMeta |          74 |     72.48 KB
gc.collect
                                                         types |   # objects |   total size
============================================================== | =========== | ============
                                      frame (codename: _leaky) |       50000 |     24.80 MB
  frame (codename: ecyglpki.Problem.intopt (ecyglpki.c:23524)) |       50000 |     19.07 MB
                                             <class 'traceback |      100000 |      6.10 MB
                                                  <class 'dict |        2856 |      2.21 MB
                                                  <class 'type |         864 |    858.94 KB
                                                  <class 'list |        5464 |    555.20 KB
                                                   <class 'set |         542 |    185.56 KB
                                                 <class 'tuple |        2348 |    155.85 KB
                                               <class 'weakref |        1632 |    127.50 KB
                                    <class 'wrapper_descriptor |        1548 |    120.94 KB
                                     <class 'method_descriptor |        1286 |     90.42 KB
                            <class 'builtin_function_or_method |        1173 |     82.48 KB
                                     <class 'getset_descriptor |        1131 |     79.52 KB
                               <class 'collections.OrderedDict |          69 |     75.62 KB
                                           <class 'abc.ABCMeta |          74 |     72.48 KB

To reproduce: python script.py the script below:

import numpy as np
import ecyglpki

nan = np.nan
nutrition_target = np.array([[ 0.57142857,  1.42857143],
   [ 0.16513761,  1.83486239],
   [ 0.57142857,  1.42857143],
   [ 0.30188679,  1.69811321],
   [ 0.90909091,         nan],
   [ 0.34586466,  1.65413534],
   [ 0.29787234,  1.70212766],
   [ 0.24175824,  1.75824176],
   [ 0.43137255,  1.56862745],
   [ 0.90909091,         nan],
   [ 0.78947368,  1.21052632],
   [ 0.90909091,         nan],
   [        nan,  2.        ],
   [ 0.0861244 ,  1.9138756 ],
   [ 0.26086957,  1.73913043],
   [ 0.90909091,         nan],
   [        nan,  2.        ],
   [ 0.90909091,         nan],
   [ 0.90909091,         nan],
   [ 0.90909091,         nan],
   [ 0.90909091,         nan],
   [ 0.02566634,  1.97433366],
   [ 0.57142857,  1.42857143],
   [        nan,  2.        ],
   [ 0.90909091,         nan],
   [ 0.90909091,         nan],
   [ 0.27160494,  1.72839506],
   [ 0.90909091,         nan],
   [ 0.90909091,         nan],
   [ 0.90909091,         nan],
   [ 0.90909091,         nan],
   [ 0.90909091,         nan],
   [ 0.98591549,  1.01408451],
   [        nan,  2.        ],
   [        nan,  2.        ],
   [        nan,  2.        ],
   [        nan,  2.        ],
   [ 0.81818182,  1.18181818],
   [ 0.72727273,  1.27272727],
   [ 0.44444444,  1.55555556],
   [ 0.66666667,  1.33333333],
   [ 0.66666667,  1.33333333],
   [        nan,  2.        ],
   [        nan,  2.        ]])

def _leaky():
    problem = ecyglpki.Problem()
    problem.add_rows(len(nutrition_target))
    problem.add_cols(20)
    
    # Configure rows/nutrients
    for i, extrema in enumerate(nutrition_target):
        problem.set_row_bnds(i+1, *extrema)
        
    # Solve
    int_opt_options = ecyglpki.IntOptControls()
    int_opt_options.presolve = True  # without this, you have to provide an LP relaxation basis
    int_opt_options.msg_lev = 'no'  # be quiet, no stdout
    try:
        problem.intopt(int_opt_options)
    except ValueError:
        pass
    
iterations = 50000

def very_leaky():
    for _ in range(iterations):
        _leaky()
    
def leaky():
    for _ in range(iterations):
        problem = ecyglpki.Problem()
        problem.add_rows(len(nutrition_target))
        problem.add_cols(20)
        
        # Configure rows/nutrients
        for i, extrema in enumerate(nutrition_target):
            problem.set_row_bnds(i+1, *extrema)
            
        # Solve
        int_opt_options = ecyglpki.IntOptControls()
        int_opt_options.presolve = True  # without this, you have to provide an LP relaxation basis
        int_opt_options.msg_lev = 'no'  # be quiet, no stdout
        try:
            problem.intopt(int_opt_options)
        except ValueError:
            pass

def normal():  # for comparison
    x=5
    while True:
        return np.ones(x)

import gc
from pympler import summary, refbrowser

sum1 = summary.summarize(gc.get_objects())
summary.print_(sum1)

very_leaky()  # leaky leaks 1 leaky frame, very_leaky leaks $iterations _leaky frames

sum1 = summary.summarize(gc.get_objects())
summary.print_(sum1)

print('gc.collect')
gc.collect()

sum1 = summary.summarize(gc.get_objects())
summary.print_(sum1)

print('showing referrers tree')
frame = next(obj for obj in gc.get_objects() if type(obj).__name__ == 'frame' and obj.f_code.co_name in ('leaky', 'very_leaky'))
refbrowser.InteractiveBrowser(frame).main()  # can take forever if iterations is set higher than 500
input()

My CPython version: 3.5.2

pip freeze (not minimal; you probably only need pympler, numpy and ecyglpki):

apipkg==1.4
attrs==16.3.0
click==6.7
colored-traceback==0.2.2
coverage==4.3.1
coverage-pth==0.0.1
decorator==4.0.10
ecyglpki==0.2.0
execnet==1.4.1
networkx==1.11
numpy==1.11.3
pandas==0.19.2
plumbum==1.6.3
py==1.4.32
Pygments==2.1.3
Pympler==0.4.3
pyprof2calltree==1.4.0
pytest==3.0.5
pytest-asyncio==0.5.0
pytest-capturelog==0.7
pytest-cov==2.4.0
pytest-env==0.6.0
pytest-localserver==0.3.6
pytest-mock==1.5.0
pytest-xdist==1.15.0
python-dateutil==2.6.0
pytz==2016.10
six==1.10.0
tabulate==0.7.7
Werkzeug==0.11.15

[equaeghe]

From your analysis, I get the impression that you may be better skilled to find a fix for this issue, certainly because I have scant time to work on ecyglpki nowadays. I would certainly do my best to integrate such a fix.