External Renderer

[odin47]

Description

Hello,

I am working on creating an external renderer to support multiple file formats(like OBJ) and displaying multiple models in a single scene using "registerRenderer" method. How do I achieve this and is there any example code regarding this.

Model Viewer Version
2.1.1

[elalish]

For this, I'd recommend you fork the repo, since three.js already has support for all this. Then you can just include their various loaders and add as many objects to our ModelScene as you want. It's nicer developing in TS anyway.

[elalish]

Yeah, that's fair. If you're willing to share your work, I could give you some help and you could add the result as a PR?

[pjdixit]

Hi Elaih,

Thank you for getting back! I was able to get it to work partially after some more fiddling. However, in AR mode I am not sure how to use my WebGL canvas to render instead of the model src attribute passed to the <model-viewer>.

I was able to get it to work to some extent in a Vue App. I will try to strip down the code to bare minimum in the next 2-3 days and send it your way.

[pjdixit]

To clarify further. here is the current live app - https://www.cadflux.com/arapp/arviewer. It uses default renderer without custom RendererInterface.

I have implemented our own non-three.js rendering engine that renders to WebGL canvas like here- https://www.cadflux.com/app/gallery.

I am trying to figure out how to use my WebGL canvas for AR through custom RendererInterface instead of default one.

[elalish]

Yes, that should be doable. I don't know of many people exercising this feature, so there may be bugs. A stripped down version should help us identify where any problems may be. Likewise, if there's anything that turns out be unclear in our docs on the subject, PRs are always welcome to improve those as well.

[pjdixit]

Its very encouraging to know it is feasible. I will send you the stripped down version soon to investigate. Thank you very much!

[pjdixit]

Hi @elalish I have attached a stripped down version of my attempt to use external renderer.

I have used a canvas slot which has canvas element with WebGL context that I would like to see in AR mode instead of the src file specified as attribute to <model-viewer> element.

I have also implemented the RendererInterface class CustomARRenderer using registerRenderer function and load, render, and resize function gets called as expected.

However, it is not clear from documentation how to specify that I would like to see my WebGL canvas element instead of GLB file in the AR mode.

This is what I see in regular mode (expected)

This what I see in AR mode ( not expected, how can I get by WebGl canvas to render here instead of the GLB file?

Source code:
ar_external_renderer.zip

Live Version - https://www.cadflux.com/Testing/ar_external_renderer/index.html

I will appreciate any guidance to achieve this. Thank you!

[elalish]

Oh, so it's working! It looks like you're just waiting for a feature we haven't implemented yet: using the external renderer for the WebXR AR session. Honestly I don't think it should be too hard - want to try a PR? I think you'll just need to copy a bit of logic from Renderer.ts to ARRenderer.ts, or maybe pack it into a function. I mostly haven't gotten around to it because building an external renderer for testing it is more work than the actual feature.

[ConfidantCommunications]

That's exactly what I am attempting to achieve. I can modify the default scene using registerEffectComposer (not using an external renderer) but it doesn't display the modified scene in AR mode.

[elalish]

Did you read my above comment? The external renderer just hasn't been hooked up to AR yet in MV. A PR would be most welcome!

[ConfidantCommunications]

Yes, thanks. I guess I am not understanding the steps it takes internally when it switches to AR mode. I can see that the constructor for ARRenderer intakes a renderer, which I assume is the default renderer. I don't want to use an external renderer if I don't need to do so. Are you saying that the only way to have ARRenderer use a runtime-modified scene is to use an external renderer? I'm willing to contribute code modifications but I want to make sure I'm going down the right path.

[ConfidantCommunications]

Okay I didn't drop off the map if you were wondering. :)

Long story but I needed to use the ExternalRenderer feature after all in order to get everything working.
My forked version of model-viewer is here.

To manipulate the Three.js scene I did this:

//canvasHolder is a reference to my <model-viewer> element.
let theCanvas = window.document.createElement("canvas");
canvasHolder.appendChild(theCanvas);
let webGLRendererParameters = { canvas : this.theCanvas, antialias : true};
let myRenderer = new CustomRenderer({ powerPreference : "high-performance"}); // I didn't do anything special in my custom class. It just extends the Three Renderer class.
canvasHolder.registerRenderer(myRenderer);
let threeWGLRenderer = new WebGLRenderer(webGLRendererParameters);
myRenderer.threeRenderer = threeWGLRenderer;

//With the reference to the scene and threeRenderer, I can add models/geometry and controls as I like. After that's done I need to call this new function in my forked model-viewer:
canvasHolder.bakeSceneToCurrentGltf(0.0254); //That number is a scaling adjustment because my geometry is using inches instead of metres.

Apologies if I missed anything above. I'm using Haxe to make this and I grabbed the compiled Javascript to summarize things.

It wasn't as simple as just drawing to the Three scene and initiating AR mode because model-viewer is designed to first load a GLTF from a file, which it keeps in memory but converts to a Three scene. When going into AR mode it needs to refer back to that stored GLTF. So the "bake" function will internally generate a new GLTF after our own scene adjustments which can then be used for the AR functions, such as generating its own USDZ for iOS.

I haven't had a chance to test this very extensively (not Android at all yet for my own technical reasons) but I wanted to share. @elalish let me know if I missed any important considerations or if you think this is suitable for a pull request.

[elalish]

Well, there are three AR modes, and they all operate very differently. The only one that goes back to the original glTF (really the URL to fetch it) is Scene Viewer on Android. WebXR uses the three.js scene directly for rendering and iOS Quick Look exports that three.js scene to a USDZ.

What change are you proposing a PR for?

[ConfidantCommunications]

I was finding that with my workflow, the internal "model" attribute in ModelScene/scene-graph used by the USDZ exporter (for iOS Quick Look) was null until I created a function to manually generate it. It doesn't seem to export directly from the live scene which I was manipulating. Normally the "model" attribute would get set when the [$onModelLoad] function was triggered, but I was not loading a GLTF model, just creating it from scratch after that event fired. My function in scene-graph.ts called [$bakeSceneToCurrentGltf] is now performing this duty for me.

[elalish]

I see - it might be easier to load a place-holder GLB, and then do your edits within model instead (which is the core of our three.js scene).

[ConfidantCommunications]

Aha, so empty the scene after the load then do the rest. I'll give that a try.