Processing time is increasing

[TheFloHub]

Hey,
Very nice tool you build here =)
I noticed the processing time for sample_sdf_near_surface is increasing for every call when I run it in a loop. Besides this it seems to work perfectly for me. It also crashes every single time in the 100th call. Is this normal? What could be the reason for this? I'm running it on windows. The mesh I'm loading here is the standford bunny.

This is my simple code:

mesh = trimesh.load(Path("D:/Code/ClassReconstruction3d/meshes/bunny.obj"), force='mesh')
mesh = scale_to_unit_sphere(mesh)
for i in range(200):
    print(i)
    startTime = time.time()
    points, sdfs = sample_sdf_near_surface(mesh, number_of_points=30000)
    print(time.time() - startTime)

Cheers,
Flo

[TheFloHub]

sorry I forgot the output^^

0
10.621185541152954
1
11.258548974990845
2
11.667204856872559
3
11.957420825958252
4
11.998848676681519
5
12.05840802192688
6
12.533770322799683
7
13.034075498580933

...
67
111.58820295333862
68
114.63963675498962
69
120.29056310653687
70
120.93348836898804
71

...

points, sdfs = sample_sdf_near_surface(mesh, number_of_points=30000)
File "D:\virtualenv\tensorflow2\lib\site-packages\mesh_to_sdf_init_.py", line 59, in sample_sdf_near_surface
surface_point_cloud = get_surface_point_cloud(mesh, surface_point_method, 1, scan_count, scan_resolution, sample_point_count, calculate_normals=sign_method=='normal' or return_gradients)
File "D:\virtualenv\tensorflow2\lib\site-packages\mesh_to_sdf_init_.py", line 17, in get_surface_point_cloud
return surface_point_cloud.create_from_scans(mesh, bounding_radius=bounding_radius, scan_count=scan_count, scan_resolution=scan_resolution, calculate_normals=calculate_normals)
File "D:\virtualenv\tensorflow2\lib\site-packages\mesh_to_sdf\surface_point_cloud.py", line 168, in create_from_scans
z_far=bounding_radius * 3
File "D:\virtualenv\tensorflow2\lib\site-packages\mesh_to_sdf\scan.py", line 61, in init
color, depth = render_normal_and_depth_buffers(mesh, camera, self.camera_transform, resolution)
File "D:\virtualenv\tensorflow2\lib\site-packages\mesh_to_sdf\pyrender_wrapper.py", line 58, in render_normal_and_depth_buffers
renderer = pyrender.OffscreenRenderer(resolution, resolution)
File "D:\virtualenv\tensorflow2\lib\site-packages\pyrender\offscreen.py", line 31, in init
self._create()
File "D:\virtualenv\tensorflow2\lib\site-packages\pyrender\offscreen.py", line 149, in create
self.platform.init_context()
File "D:\virtualenv\tensorflow2\lib\site-packages\pyrender\platforms\pyglet_platform.py", line 52, in init_context
width=1, height=1)
File "D:\virtualenv\tensorflow2\lib\site-packages\pyglet\window\win32_init.py", line 134, in init
super(Win32Window, self).init(*args, **kwargs)
File "D:\virtualenv\tensorflow2\lib\site-packages\pyglet\window_init.py", line 648, in init
self.create()
File "D:\virtualenv\tensorflow2\lib\site-packages\pyglet\window\win32_init.py", line 285, in create
self.switch_to()
File "D:\virtualenv\tensorflow2\lib\site-packages\pyglet\window\win32_init.py", line 348, in switch_to
self.context.set_current()
File "D:\virtualenv\tensorflow2\lib\site-packages\pyglet\gl\win32.py", line 243, in set_current
super(Win32Context, self).set_current()
File "D:\virtualenv\tensorflow2\lib\site-packages\pyglet\gl\base.py", line 334, in set_current
gl_info.set_active_context()
File "D:\virtualenv\tensorflow2\lib\site-packages\pyglet\gl\gl_info.py", line 91, in set_active_context
self.vendor = asstr(cast(glGetString(GL_VENDOR), c_char_p).value)
File "D:\virtualenv\tensorflow2\lib\site-packages\pyglet\gl\lib.py", line 107, in errcheck
raise GLException(msg)
pyglet.gl.lib.GLException: b'Der Vorgang ist ung\xfcltig.'

[GTO2013]

I am not 100% sure this is the problem, but for every scan this is called, which creates a new scene with the mesh and camera parameters for every scan. I am not sure if this gets cleaned up properly. It would probably be better to setup the scene with the mesh and camera once and just update the camera pose for each scan.

def render_normal_and_depth_buffers(mesh, camera, camera_transform, resolution):
    global suppress_multisampling
    suppress_multisampling = True
    scene = pyrender.Scene()
    scene.add(pyrender.Mesh.from_trimesh(mesh, smooth = False))
    scene.add(camera, pose=camera_transform)

    renderer = pyrender.OffscreenRenderer(resolution, resolution)
    renderer._renderer._program_cache = CustomShaderCache()

    color, depth = renderer.render(scene, flags=pyrender.RenderFlags.SKIP_CULL_FACES)
    suppress_multisampling = False
    return color, depth

[liangpan99]

I am not 100% sure this is the problem, but for every scan this is called, which creates a new scene with the mesh and camera parameters for every scan. I am not sure if this gets cleaned up properly. It would probably be better to setup the scene with the mesh and camera once and just update the camera pose for each scan.

def render_normal_and_depth_buffers(mesh, camera, camera_transform, resolution):
    global suppress_multisampling
    suppress_multisampling = True
    scene = pyrender.Scene()
    scene.add(pyrender.Mesh.from_trimesh(mesh, smooth = False))
    scene.add(camera, pose=camera_transform)

    renderer = pyrender.OffscreenRenderer(resolution, resolution)
    renderer._renderer._program_cache = CustomShaderCache()

    color, depth = renderer.render(scene, flags=pyrender.RenderFlags.SKIP_CULL_FACES)
    suppress_multisampling = False
    return color, depth

I have tested and found that the renderer cannot be cleaned up properly. When the function render_normal_and_depth_buffers is done, the clean up process will get stucked at this position (see the image) in file pyrender/renderer.py. Because the mesh_to_sdf use a custom shader and it doesn't define a clear() function.

To clean up properly, i add a clear() function like this:

# Render a normal buffer instead of a color buffer
class CustomShaderCache():
    def __init__(self):
        self.program = None

    def get_program(self, vertex_shader, fragment_shader, geometry_shader=None, defines=None):
        if self.program is None:
            shaders_directory = os.path.join(os.path.dirname(__file__), 'shaders')
            self.program = pyrender.shader_program.ShaderProgram(os.path.join(shaders_directory, 'mesh.vert'), os.path.join(shaders_directory, 'mesh.frag'), defines=defines)
        return self.program
   
   # For clean up
   def clear(self):
        self.program.delete()
        self.program = None

Hope it works.