There is a gap in metrics between the trained and pretrained models.

[czc2000]

I tried to train on the n3d dataset, and although the render results are very good, the metrics are a bit off compared to the trained model, for example in the coffee_martini scene my average psnr is only 28.13 while the trained model reaches 28.73, and there are similar losses of performance in the other scenes.
Two results are attached here, the first is the trained coffee-martini and the second is my trained coffee-martini.
I used the default settings for training and I'm thinking about what is not set right, can you help me?

{
 "SSIM": 0.918133020401001,
 "SKSSIM": 0.9115921258926392,
 "SKSSIM2": 0.950756311416626,
 "PSNR": 28.728668212890625,
 "LPIPS": 0.07040495425462723,
 "LPIPSVGG": 0.15612900257110596,
 "times": 0.026875119656324387
}

{
 "SSIM": 0.9155685305595398,
 "SKSSIM": 0.9085630178451538,
 "SKSSIM2": 0.9487462639808655,
 "PSNR": 28.13457679748535,
 "LPIPS": 0.07727879285812378,
 "LPIPSVGG": 0.1607026904821396,
 "times": 0.026210365816950798
}

[juno181]

Thanks for verification. This is because the last iteration was not automatically added to save_iterations. I've fixed my code for this. Please check the updated code.

[czc2000]

Thanks for your reply !

[czc2000]

Thank you very much for the update. I updated the code and re-trained and found that the gap didn't get a good improvement, and there is a general 0.4 to 1.1 drop in PSNR per scene compared to the pre-trained results.The performance degradation seems to be outside the margin of error. In the scenarios cut_roasted_beef, cook_spinach, the drop is more pronounced
And I was looking at the training logs and noticed that the psnr drop sometimes occurs from 30000 to 40000 steps, i.e. the PSNR of the test performed at 30000 steps will be higher than 40000 steps, I would like to ask if this is normal?
I noticed that there is no fixed random seed in the code, is it because of the effect of the seed? Or is there any other setting differences between the current default settings and the pre-trained model? I checked their cfg_arg files and found that the parameters are mostly the same.
I'm excited about your work and looking forward to your reply！

Thanks for verification. This is because the last iteration was not automatically added to save_iterations. I've fixed my code for this. Please check the updated code.

[juno181]

In my opinion, a general 1 drop in PSNR is unusual. When I reproduced it, I got slightly better results in the coffee martini scene. The PSNR generally increased from step 30,000 to step 40,000. Also, while the seed does have an effect, cases where it causes a change of more than 0.5 in PSNR were very rare. Have you used configs/N3V/n3v_base.json? It is recommended to use this config when training a 3D video neural dataset. To use it, simply add the argument --config configs/N3V/n3v_base.json.

{
 "SSIM": 0.9186709523200989,
 "SKSSIM": 0.9123628735542297,
 "SKSSIM2": 0.9508540034294128,
 "PSNR": 28.865598678588867,
 "LPIPS": 0.07560530304908752,
 "LPIPSVGG": 0.1579393446445465,
 "times": 0.016616083681583405
}

[czc2000]

I have added the argument --config configs/N3V/n3v_base.json before.And I double checked the config settings. Do you think there are any other factors that could be making the results abnormal, such as the python environment or dataset preprocess error? But I have followed your documentation to set up the environment and preprocess the dataset.

[czc2000]

I attach the results of my training here.There are some scene where the psnr drop is small, but generally did not reach the pretrain model.
And my command use like this
python train.py --config configs/N3V/n3v_base.json --model_path ./output/dynerf/coffee_martini --source_path /N3D/coffee_martini

## coffee martini
{
 "SSIM": 0.9139772653579712,
 "SKSSIM": 0.9069717526435852,
 "SKSSIM2": 0.9472635984420776,
 "PSNR": 28.070709228515625,
 "LPIPS": 0.07867260277271271,
 "LPIPSVGG": 0.16166387498378754,
 "times": 0.029493017122149467
}
## cook_spinach
{
 "SSIM": 0.9547927975654602,
 "SKSSIM": 0.9501733183860779,
 "SKSSIM2": 0.9747482538223267,
 "PSNR": 32.187156677246094,
 "LPIPS": 0.04555555433034897,
 "LPIPSVGG": 0.14064113795757294,
 "times": 0.019486475735902786
}
## cut_roasted_beef
{
 "SSIM": 0.9573107957839966,
 "SKSSIM": 0.9529886841773987,
 "SKSSIM2": 0.9769240021705627,
 "PSNR": 33.306461334228516,
 "LPIPS": 0.04487214609980583,
 "LPIPSVGG": 0.13619321584701538,
 "times": 0.02090240642428398
}
## flame_salmon_1
{
 "SSIM": 0.9216399192810059,
 "SKSSIM": 0.9152145385742188,
 "SKSSIM2": 0.952958345413208,
 "PSNR": 28.873685836791992,
 "LPIPS": 0.0703233927488327,
 "LPIPSVGG": 0.1495954841375351,
 "times": 0.029643261805176735
}

## flame_steak
{
 "SSIM": 0.9619386792182922,
 "SKSSIM": 0.958199143409729,
 "SKSSIM2": 0.9797468185424805,
 "PSNR": 33.531524658203125,
 "LPIPS": 0.036806341260671616,
 "LPIPSVGG": 0.12295567989349365,
 "times": 0.0215836800634861
}
##sear_steak
{
 "SSIM": 0.961395263671875,
 "SKSSIM": 0.9577805399894714,
 "SKSSIM2": 0.9793656468391418,
 "PSNR": 32.8525276184082,
 "LPIPS": 0.038309693336486816,
 "LPIPSVGG": 0.12698908150196075,
 "times": 0.02232000231742859
}

[czc2000]

I checked further and found my input.ply file, I found some discrepancies between the number of points in my initialised point cloud and the number of points in the pre-training model, for example in the coffee_martini scenario, the number of points in my input.ply is 4945, while the number of points in the pre-training model is 5498. do you think that the this initialisation gap will lead to a gap in the final result?
If it does have an effect, how should I get the correct initialised point cloud, and is it related to the version of colmap? Currently I'm using your data preprocessing scripts preprocess_all_n3v.sh.

[juno181]

I appreciate the detailed check. I agree with your observations. In my experiments, I used a sparse point cloud, which might have left some parts of the point cloud uninitialized. In the case of the coffee martini scene, most of the PSNR loss occurred in areas where the point cloud wasn’t well-initialized. This corresponds to the background area in the attached image. I believe this issue shares similarities with random initialization in Gaussian Splatting, which could lead to a performance gap. To address the initialization issue, the point cloud needs to be densified. COLMAP supports densification using MVS. 4DGaussians also uses this approach, and it may be helpful to refer to the 4DGaussians for more information. I will revise the preprocessing to ensure stable training.

[czc2000]

I appreciate the detailed check. I agree with your observations. In my experiments, I used a sparse point cloud, which might have left some parts of the point cloud uninitialized. In the case of the coffee martini scene, most of the PSNR loss occurred in areas where the point cloud wasn’t well-initialized. This corresponds to the background area in the attached image. I believe this issue shares similarities with random initialization in Gaussian Splatting, which could lead to a performance gap. To address the initialization issue, the point cloud needs to be densified. COLMAP supports densification using MVS. 4DGaussians also uses this approach, and it may be helpful to refer to the 4DGaussians for more information. I will revise the preprocessing to ensure stable training.

Thank you very much for your reply. I have tried several trains, and I found that the current training seems to be rather unstable, the difference between the best case and the worst case PSNR will be close to 1. But only very few results can be close to the results of the pre-trained model.
By the way,I also found that my file size seems to be bigger than the pre-training file, for example, the pre-training model is about 130M while my model is about 160M, and I found that this difference is mainly in dynamic_point_cloud.ply
This is strange, although PSNR may be affected by random initialisation, I'm not sure if this makes such a big difference in file size with the same settings.

## pretrain
total 133081
60101720 Aug  4 05:14 dynamic_point_cloud.ply
75971269 Aug  4 05:14 point_cloud.ply
## my train
total 167921
96234943 Nov 12 00:48 dynamic_point_cloud.ply
75410189 Nov 12 00:48 point_cloud.ply

Waiting for your update.

[juno181]

I think this issue is also related to random initialization because some "dead Gaussians" are not entirely pruned. As far as I know, with around 30MB, there is a difference of approximately 25,000 Dynamic Gaussians, and I believe that splitting is also incorrectly handled due to improper initialization. I expect that a well-initialized point cloud could help mitigate this phenomenon.

[czc2000]

I checked the dynamic_point_cloud.ply of the pre-trained model and my trained model and found some unusual differences, in my dynamic point cloud my motion_xyz is only 36 time points, but in the pre-trained point cloud there are 37 time points。motion_rot is the same as motion_xyz,one is 36 , another is 37.
I've attached the headers for both files, mine on top and the pre-trained model below. This issue seems to cause me to have 0 in the last time dimension of motion_xyz after the load_ply function is done.
Does this mean that there are some differences in the process of time expanding during the training of the two models? Maybe the error comes from this difference
Looking forward to your reply.

ply
format binary_little_endian 1.0
element vertex 71770
property float motion_xyz_0_0
property float motion_xyz_0_1
property float motion_xyz_0_2
property float motion_xyz_1_0
property float motion_xyz_1_1
property float motion_xyz_1_2
property float motion_xyz_2_0
property float motion_xyz_2_1
property float motion_xyz_2_2
property float motion_xyz_3_0
property float motion_xyz_3_1
property float motion_xyz_3_2
property float motion_xyz_4_0
property float motion_xyz_4_1
property float motion_xyz_4_2
property float motion_xyz_5_0
property float motion_xyz_5_1
property float motion_xyz_5_2
property float motion_xyz_6_0
property float motion_xyz_6_1
property float motion_xyz_6_2
property float motion_xyz_7_0
property float motion_xyz_7_1
property float motion_xyz_7_2
property float motion_xyz_8_0
property float motion_xyz_8_1
property float motion_xyz_8_2
property float motion_xyz_9_0
property float motion_xyz_9_1
property float motion_xyz_9_2
property float motion_xyz_10_0
property float motion_xyz_10_1
property float motion_xyz_10_2
property float motion_xyz_11_0
property float motion_xyz_11_1
property float motion_xyz_11_2
property float motion_xyz_12_0
property float motion_xyz_12_1
property float motion_xyz_12_2
property float motion_xyz_13_0
property float motion_xyz_13_1
property float motion_xyz_13_2
property float motion_xyz_14_0
property float motion_xyz_14_1
property float motion_xyz_14_2
property float motion_xyz_15_0
property float motion_xyz_15_1
property float motion_xyz_15_2
property float motion_xyz_16_0
property float motion_xyz_16_1
property float motion_xyz_16_2
property float motion_xyz_17_0
property float motion_xyz_17_1
property float motion_xyz_17_2
property float motion_xyz_18_0
property float motion_xyz_18_1
property float motion_xyz_18_2
property float motion_xyz_19_0
property float motion_xyz_19_1
property float motion_xyz_19_2
property float motion_xyz_20_0
property float motion_xyz_20_1
property float motion_xyz_20_2
property float motion_xyz_21_0
property float motion_xyz_21_1
property float motion_xyz_21_2
property float motion_xyz_22_0
property float motion_xyz_22_1
property float motion_xyz_22_2
property float motion_xyz_23_0
property float motion_xyz_23_1
property float motion_xyz_23_2
property float motion_xyz_24_0
property float motion_xyz_24_1
property float motion_xyz_24_2
property float motion_xyz_25_0
property float motion_xyz_25_1
property float motion_xyz_25_2
property float motion_xyz_26_0
property float motion_xyz_26_1
property float motion_xyz_26_2
property float motion_xyz_27_0
property float motion_xyz_27_1
property float motion_xyz_27_2
property float motion_xyz_28_0
property float motion_xyz_28_1
property float motion_xyz_28_2
property float motion_xyz_29_0
property float motion_xyz_29_1
property float motion_xyz_29_2
property float motion_xyz_30_0
property float motion_xyz_30_1
property float motion_xyz_30_2
property float motion_xyz_31_0
property float motion_xyz_31_1
property float motion_xyz_31_2
property float motion_xyz_32_0
property float motion_xyz_32_1
property float motion_xyz_32_2
property float motion_xyz_33_0
property float motion_xyz_33_1
property float motion_xyz_33_2
property float motion_xyz_34_0
property float motion_xyz_34_1
property float motion_xyz_34_2
property float motion_xyz_35_0
property float motion_xyz_35_1
property float motion_xyz_35_2

ply
format binary_little_endian 1.0
element vertex 47692
property float motion_xyz_0_0
property float motion_xyz_0_1
property float motion_xyz_0_2
property float motion_xyz_1_0
property float motion_xyz_1_1
property float motion_xyz_1_2
property float motion_xyz_2_0
property float motion_xyz_2_1
property float motion_xyz_2_2
property float motion_xyz_3_0
property float motion_xyz_3_1
property float motion_xyz_3_2
property float motion_xyz_4_0
property float motion_xyz_4_1
property float motion_xyz_4_2
property float motion_xyz_5_0
property float motion_xyz_5_1
property float motion_xyz_5_2
property float motion_xyz_6_0
property float motion_xyz_6_1
property float motion_xyz_6_2
property float motion_xyz_7_0
property float motion_xyz_7_1
property float motion_xyz_7_2
property float motion_xyz_8_0
property float motion_xyz_8_1
property float motion_xyz_8_2
property float motion_xyz_9_0
property float motion_xyz_9_1
property float motion_xyz_9_2
property float motion_xyz_10_0
property float motion_xyz_10_1
property float motion_xyz_10_2
property float motion_xyz_11_0
property float motion_xyz_11_1
property float motion_xyz_11_2
property float motion_xyz_12_0
property float motion_xyz_12_1
property float motion_xyz_12_2
property float motion_xyz_13_0
property float motion_xyz_13_1
property float motion_xyz_13_2
property float motion_xyz_14_0
property float motion_xyz_14_1
property float motion_xyz_14_2
property float motion_xyz_15_0
property float motion_xyz_15_1
property float motion_xyz_15_2
property float motion_xyz_16_0
property float motion_xyz_16_1
property float motion_xyz_16_2
property float motion_xyz_17_0
property float motion_xyz_17_1
property float motion_xyz_17_2
property float motion_xyz_18_0
property float motion_xyz_18_1
property float motion_xyz_18_2
property float motion_xyz_19_0
property float motion_xyz_19_1
property float motion_xyz_19_2
property float motion_xyz_20_0
property float motion_xyz_20_1
property float motion_xyz_20_2
property float motion_xyz_21_0
property float motion_xyz_21_1
property float motion_xyz_21_2
property float motion_xyz_22_0
property float motion_xyz_22_1
property float motion_xyz_22_2
property float motion_xyz_23_0
property float motion_xyz_23_1
property float motion_xyz_23_2
property float motion_xyz_24_0
property float motion_xyz_24_1
property float motion_xyz_24_2
property float motion_xyz_25_0
property float motion_xyz_25_1
property float motion_xyz_25_2
property float motion_xyz_26_0
property float motion_xyz_26_1
property float motion_xyz_26_2
property float motion_xyz_27_0
property float motion_xyz_27_1
property float motion_xyz_27_2
property float motion_xyz_28_0
property float motion_xyz_28_1
property float motion_xyz_28_2
property float motion_xyz_29_0
property float motion_xyz_29_1
property float motion_xyz_29_2
property float motion_xyz_30_0
property float motion_xyz_30_1
property float motion_xyz_30_2
property float motion_xyz_31_0
property float motion_xyz_31_1
property float motion_xyz_31_2
property float motion_xyz_32_0
property float motion_xyz_32_1
property float motion_xyz_32_2
property float motion_xyz_33_0
property float motion_xyz_33_1
property float motion_xyz_33_2
property float motion_xyz_34_0
property float motion_xyz_34_1
property float motion_xyz_34_2
property float motion_xyz_35_0
property float motion_xyz_35_1
property float motion_xyz_35_2
property float motion_xyz_36_0
property float motion_xyz_36_1
property float motion_xyz_36_2

[Drow999]

Thank you very much for the update. I updated the code and re-trained and found that the gap didn't get a good improvement, and there is a general 0.4 to 1.1 drop in PSNR per scene compared to the pre-trained results.The performance degradation seems to be outside the margin of error. In the scenarios cut_roasted_beef, cook_spinach, the drop is more pronounced And I was looking at the training logs and noticed that the psnr drop sometimes occurs from 30000 to 40000 steps, i.e. the PSNR of the test performed at 30000 steps will be higher than 40000 steps, I would like to ask if this is normal? I noticed that there is no fixed random seed in the code, is it because of the effect of the seed? Or is there any other setting differences between the current default settings and the pre-trained model? I checked their cfg_arg files and found that the parameters are mostly the same. I'm excited about your work and looking forward to your reply！

Thanks for verification. This is because the last iteration was not automatically added to save_iterations. I've fixed my code for this. Please check the updated code.

Yes, I also got 28.03 PSNR in coffee martini scene with 40K iteration training, 28.16 with 30K iteration.

[juno181]

I checked the dynamic_point_cloud.ply of the pre-trained model and my trained model and found some unusual differences, in my dynamic point cloud my motion_xyz is only 36 time points, but in the pre-trained point cloud there are 37 time points。motion_rot is the same as motion_xyz,one is 36 , another is 37. I've attached the headers for both files, mine on top and the pre-trained model below. This issue seems to cause me to have 0 in the last time dimension of motion_xyz after the load_ply function is done. Does this mean that there are some differences in the process of time expanding during the training of the two models? Maybe the error comes from this difference Looking forward to your reply.

This result was generated by reducing keyframe values during code cleaning. I intentionally increased the values to mitigate issues arising when loading files. The final timestamp does not affect performance, as it is not actually used in rendering. I will continue to track this issue.

[NeutrinoLiu]

also want to ask if anyone has reproduced the 33.2 PSNR for cook_spinach scene? the best i could get is only a little more than 32. In my rendering result, there are obvious artifact for the hand region where high dynamics is indicated. most other 4DGS works also did a bad work on this region yet ex4dgs seems more worse in such region.

[czc2000]

also want to ask if anyone has reproduced the 33.2 PSNR for cook_spinach scene? the best i could get is only a little more than 32. In my rendering result, there are obvious artifact for the hand region where high dynamics is indicated. most other 4DGS works also did a bad work on this region yet ex4dgs seems more worse in such region.

I got the same result as yours