Overactive Python and Java GCs

[jazznoodler]

I have some python code running on Linux where the JPype GC algo seems to trigger a dance between the 2 GC mechanisms (3 or 4 gcs each) before it settles down again. Is there a way to debug this? Any ideas what might be happening ? Thanks

[Thrameos]

Generally speaking one GC will trigger the other. Though Java ones should be much more infrequent than Python ones. There are statistical triggers in the native/common/jp_gc file that you can instrument to see which high water mark is triggering the other. To get multiple triggers in a row you would have to be in the condition where one or both virtual machines were nearing exhaustion which would trigger a call to the other to free memory.

[jazznoodler]

Hi you may find this useful or not: took a look at the code and the GC interactions unfortunately the watermark algo appears to be broken on Linux for some reason the current memory value calc is out by a approx a factor of 10 leading to a lot unnecessary GC cycles in our process.
On Windows our slowly growing python heap 5GB and static Java heap leads to triggering the predict part of the algo many times - plotting maximum and minimum we get sawtooth with the high and low watermarks converging..we get about 250 unnecessary cycles .Disabling the external system.GC call is a workaround on both platforms with the caveat we won't have Java proxies to large python objects, Also the JPype GC callback is added twice but removing the redundant one doesn't help.

[Thrameos]

It is possible there is a unit error which which some flavors of linux report their memory in different ways which may lead to the factor of 10 issue. If we could identify the unit properly and fix it then that would help.

With regard to selecting the GC behavior, if you want to submit a patch in which you programmatically disable the GC it would be fine. Or just expose the watermarks such that they can be adjusted by the user. Though this may be a bit much to ask. I am unfortunately too busy with work to put much time into JPype.

[jazznoodler]

Hey Karl thanks for the responsei will try and make some time to add an option to turn off the external System GC callcheers On Monday, 13 February 2023 at 19:33:43 GMT, Karl Nelson ***@***.***> wrote: It is possible there is a unit error which which some flavors of linux report their memory in different ways which may lead to the factor of 10 issue. If we could identify the unit properly and fix it then that would help. With regard to selecting the GC behavior, if you want to submit a patch in which you programmatically disable the GC it would be fine. Or just expose the watermarks such that they can be adjusted by the user. Though this may be a bit much to ask. I am unfortunately too busy with work to put much time into JPype. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>