Optimize Lazy Camera Loading for Efficient Compression of Large Files with Numerous Images and Corresponding Cameras (#8)

[EtaCassiopeia]

• Implemented LazyCameraLoader class in camera_utils.py to enable on-demand loading and unloading of cameras.
• Refactored calc_importance function to use LazyCameraLoader for efficient memory management.
• Modified finetune function to leverage LazyCameraLoader and ensure memory is freed after processing each camera.
• Improved memory usage and performance during sensitivity calculation and finetuning processes.
• This solution is particularly beneficial for handling large files and datasets, though it may not be as efficient for smaller datasets.

This commit addresses issue #8.

[arcman7]

LGTM

[bzy-080408]

I tried but sttill out of memory. gaussian splatting model used about 4000x 4k photo.my device has 660g ram and 3x tesla l20 48Gb vram. date put into cpu memory

[arcman7]

Each image was 4MB and you had 4000 of them? How many splats were in your scene when it crashed?

[bzy-080408]

Each image was 4MB and you had 4000 of them? How many splats were in your scene when it crashed?

[bzy-080408]

4396589,this is the pointcloud file before compression

[bzy-080408]

I tried by a 1.1TB RAM device.still out of memory

[arcman7]

I need to dig up the code again and take a look. I know we had this issue fixed. Is this a time sensitive issue for you?

[bzy-080408]

It`s all the time,and anyway to fix it?

…

________________________________ 发件人: Andrew Ryan Carpenter ***@***.***> 发送时间: 2024年11月12日 9:15 收件人: KeKsBoTer/c3dgs ***@***.***> 抄送: 白致远 ***@***.***>; Comment ***@***.***> 主题: Re: [KeKsBoTer/c3dgs] Optimize Lazy Camera Loading for Efficient Compression of Large Files with Numerous Images and Corresponding Cameras (#8) (PR #9) I need to dig up the code again and take a look. I know we had this issue fixed. Is this a time sensitive issue for you? ― Reply to this email directly, view it on GitHub<#9 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APFN7UP7KXUIPMUXS7G3P7D2AFJE3AVCNFSM6AAAAABKDWVYWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRZGM4TAMRRHA>. You are receiving this because you commented.

[bzy-080408]

I need to dig up the code again and take a look. I know we had this issue fixed. Is this a time sensitive issue for you?

so anyway to fix it?

[arcman7]

Use something like this:

def lazy_call(f, *args, **kwargs):
    return lambda: f(*args, **kwargs)
    
def cameraList_from_camInfos(cam_infos, resolution_scale, args, lazy_load):
    camera_list = []

    # for id, c in enumerate(cam_infos):
    #     camera_list.append(loadCam(args, id, c, resolution_scale))
    
    for id, c in enumerate(cam_infos):
        camera_list.append(loadCam(args, id, c, resolution_scale))
        if lazy_load:
            camera_list.append(lazy_call(loadCam, args, id, c, resolution_scale))
        else:
            camera_list.append(loadCam(args, id, c, resolution_scale))

    return camera_list