loss item

[roywithfiringblade]

Thanks for sharing this work. i am a little confused about the loss item. i know that 'coarse_raycolor' is corresponding to L_render in paper, what is 'ray_masked_loss' and 'ray_miss_loss'?
And if i understand correctly, 'zero_one_loss_items' is the L_sparse in paper. but since we already set the neural points as input data and didn't load mvsnet model, can the parameters of mvsnet also be updated? (when i inspect the variable , the only model is "ray_marching") And you set the neural points to nn.parameter to update, any reasons behind it?

[Xharlie]

hi the ray_masked_loss is the loss that only apply to the rays which have neural points neighbors, this is the real loss that drive the opitmization, coarse_raycolor is the ray_masked_loss + rays that hit nothing and have no gradient. ray_miss_loss is the metric to see whether the ray hit no neural points actually have foreground color.
please see the "color loss weight", if a loss has weight 0.0, it means it just for better understanding, not for optimization

[roywithfiringblade]

thanks for the explanation, can you explain a bit about what is the purpose of parameterizing the neural points?

…

---Original--- From: "Qiangeng ***@***.***> Date: Tue, Mar 22, 2022 23:18 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [Xharlie/pointnerf] loss item (Issue #16) hi the ray_masked_loss is the loss that only apply to the rays which have neural points neighbors, this is the real loss that drive the opitmization, coarse_raycolor is the ray_masked_loss + rays that hit nothing and have no gradient. ray_miss_loss is the metric to see whether the ray hit no neural points actually have foreground color. please see the "color loss weight", if a loss has weight 0.0, it means it just for better understanding, not for optimization — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[roywithfiringblade]

@Xharlie Sorry, stille confused about the neural points parameterization, can you reopen the issue?

[Xharlie]

hi what do you mean by the "neural points parameterization", i guess you are confused about the input to compute radiance?
we use the world coordinate distance, camera coordinate distance between a neural point to shading location, and point feature to calculate the contribution from neural point to shading location, then send the weighted sum inter feature to calculate color and density.

[roywithfiringblade]

@Xharlie in neural_points.py :code likeself.xyz = nn.Parameter(saved_features["neural_points.xyz"]) and self.points_embeding = nn.Parameter(points_embeding) turns the input to parameters.
and in mvs_points_volumetric_model.py line 87 :
if len(neural_params) > 0: self.neural_point_optimizer = torch.optim.Adam(neural_params, lr=opt.plr, #/ 5.0, betas=(0.9, 0.999)) self.optimizers.append(self.neural_point_optimizer)
you use an adam optimizer to update the neural points directly, rather than optimize the mvs model. That is what i am confused.
Cause the neural points should be the output of the pre-trained mvsnet, and it seems we only optimize the mlp model and the neural points direclty, the mvs model is fixed (after dtu dataset pretrained).

[Xharlie]

yea, the mvs model is fixed, since later we found using the pretrained mvs model is already good enough

[roywithfiringblade]

@Xharlie if the mvs model is already fixed, the confidence of every point should also be unchanged. so how does point pruning works?

[Xharlie]

hi, i think you confused point initialization (mvs, etc.) with per-scene optimization, please refer to Fig.3 in the paper. All the details are in the paper, i have to retype my paper here to answer this question.

[roywithfiringblade]

hi, i think you confused point initialization (mvs, etc.) with per-scene optimization, please refer to Fig.3 in the paper. All the details are in the paper, i have to retype my paper here to answer this question.

Thanks for explanation, i think i have understood that. @Xharlie