Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Dsine Surface Normal Option #17

Merged
merged 11 commits into from
Apr 17, 2024
Merged

Add Dsine Surface Normal Option #17

merged 11 commits into from
Apr 17, 2024

Conversation

pablovela5620
Copy link
Contributor

@pablovela5620 pablovela5620 commented Apr 16, 2024

Add the option to use the DSINE surface normal estimator

To test out, just run

pixi run example-polycam

This will

  1. Download an example polycam scan
  2. Convert said polycam scan -> nerfstudio format
  3. Generate surface normals based on DSINE
  4. Create a pointcloud similar to how its done for mushroom
  5. Train using dnsplatter
  • Included a script to compare different surface normal estimations. Can be run by
python dn_splatter/scripts/compare_normals.py \
   $PATH TO FIRST SURFACE NORMAL DIR \
   $PATH TO SCOND SURFACE NORMAL DIR

Heres an example of a custom dataset I collected
custom_dsine_v_omni

  • Estimate surface normals from pretrain using
python dn_splatter/scripts/normals_from_pretrain.py --model-type dsine

DSINE predicts normals in a range of -1 to 1 in camera coordinate system LUB (+X left | +Y up | +Z Back), I convert them to RUB.
I left the current convention names as they are but they appear to be backwards (no issues with functionality, just naming convention errors #9

  • Also added a small change to mushroom pointclouds utils to allow generation of pointclouds for polycam datasets generated by nerfstudio, this allows for using datasets generated via polycam to not have to use random gaussian initialization which leads to better convergence
python dn_splatter/data/mushroom_utils/pointcloud_utils.py
  • added normal visualization and metrics to get_image_metrics_and_images

add dsine surface normal

add normal comparison visualization

update pixi.toml

include geffnet dependancy for dsine

Add surface normal comparison visualization

add icecream dep to pixi and caching for pip install -e

add normal metrics/images to get_img_metrics

fix pyproject bug

Update pixi.toml with new dependencies and outputs cache

add coordinate system conversion for DSINE normal network

add function to generate pointcloud for polycam data processed by ns-process-data

add train polycam command to pixi

Update pixi.toml with gcc and gxx

Update pixi.toml with Quick Run example and other training examples
@pablovela5620 pablovela5620 changed the title (WIP) Add Dsine Surface Normal Option Add Dsine Surface Normal Option Apr 17, 2024
maturk and others added 3 commits April 17, 2024 17:34
@maturk maturk merged commit 8d6cce6 into maturk:main Apr 17, 2024
@pablovela5620 pablovela5620 deleted the dsine branch April 17, 2024 15:03
@gaetan-landreau
Copy link

Hi ! 👋🏼
Regarding the DSINE normal orientation convention, this issue seems to admit a RDF convention (it's equivalent, RDF = -LUB). But then the conversion to RUB through diag(-1,1,1) seems a bit wrong to me then. Am I missing something ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants