Add model dragging and resizing in XR headsets by controllers or hands

[FluorescentHallucinogen]

This pull request fixes a problem (described by me a year ago in the #3969 (reply in thread)) that users can't move and rotate models in the Meta Quest headsets.

[elalish]

Thanks, this is exciting! Now I just need to hunt down a Quest so I can test this personally...

[FluorescentHallucinogen]

@elalish What are the chances of you getting your hands on Meta Quest? I thought it was a must have for the model-viewer maintainer. :)

[elalish]

Well, now that we're looking to get serious about adding headset support, I'd agree. I'm going to get one ordered now.

[FluorescentHallucinogen]

we're looking to get serious about adding headset support

That's great news!

I'm going to get one ordered now.

Please let me know once you order the headset and which model (Meta Quest 3 or Meta Quest Pro). And when you get it in your hands. ;)

I expect when we add support for XR headsets and test the code on Meta Quest, we will automatically have Apple Vision Pro support since it's the same WebXR standard here and there. :)

[adamwolf]

(Is there something here that other folks could help along?)

[elalish]

@adamwolf UX design, research, or just musings/examples would all help at this stage. It's harder to figure out what to build than to build it.

[elalish]

I know this has been on the backburner for way too long, but I do finally have a Quest 3 on my desk and I just tested this PR in it. Good start! Got a ways to go, but no problem there. Now to do some actual coding...

[elalish]

I'm actually decently happy with this UX now - would love to hear some feedback from you all. There are two main things I dislike still:

1. The model's initial placement is too far away. This should be an easy fix; it's simply because the FoV of a headset is way wider than a phone camera.
2. It's difficult to turn the model around its vertical axis. This is mostly just an XR controller ergonomics problem, but we need a way to work around it since that's the primary way you want to reorient, say, furniture. I'm hoping some tricky matrix math can improve this.

[FluorescentHallucinogen]

@elalish Thanks for your help! Just tested. LGTM. I've removed the "Draft" label. Let's merge it. Further UX improvements can be done by separate pull requests.

[FluorescentHallucinogen]

The model's initial placement is too far away.

It seems this depends on the 3D model (its size?). E.g. small models https://modelviewer.dev/assets/ShopifyModels/Mixer.glb and https://modelviewer.dev/assets/ShopifyModels/GeoPlanter.glb models are initially pretty close, but big https://modelviewer.dev/shared-assets/models/NeilArmstrong.glb and https://modelviewer.dev/assets/ShopifyModels/Chair.glb models are initially far from the user.

It's difficult to turn the model around its vertical axis. This is mostly just an XR controller ergonomics problem

This is a current implementation problem. I see 2 problems:

1. You can only select the model only using the bottom plate (transparent rectangle with white edges, which becomes visible only by pointing the ray at the bottom of the model). It's not very intuitive. I originally tried to grab an random point at the top of the model. And it didn't work. And at this time, the bottom plate was not visible at all.

2. It's impossible to rotate and leave the model the way I want it. E.g., you cannot turn the model upside down or lay it on its side. It keeps turning bottom down.

[elalish]

You can only select the model only using the bottom plate

Agreed, forgot to mention I'll switch this to grabbing the whole bounding volume.

It's impossible to rotate and leave the model the way I want it. E.g., you cannot turn the model upside down or lay it on its side. It keeps turning bottom down.

This is by design - before it was impossible to get it to sit upright on the floor once selected, which I think is the primary use case. This is also why I'm going to work on this a bit more before merging. Once someone is used to one UX, any change (that's not a pure addition) is seen as worse, due to anchoring bias.

I'm pretty happy with inspectability, especially now that we can pass the object back and forth between the two controllers/hands. If you have a specific idea for improving the UX, I'll be happy to give it a try.

[FluorescentHallucinogen]

it was impossible to get it to sit upright on the floor once selected, which I think is the primary use case
I'm pretty happy with inspectability

I'm not sure that's really the primary use case.

Look at all these cool demos/promos for Apple Vision Pro. E.g., kids leave in the air some 3D model such as a human heart or a complex animated mechanism with gears in their desired position, and they can walk around it and look at it from all sides, including from the bottom.

In the current implementation:

1. the 3D model is constantly turns bottom down
2. the 3D model constantly falls to the floor

For me, the current implementation is only good for large objects like furniture.

It's not very convenient to view small 3D models like a mixer if they fall on the floor. :-D

Yes, you can still view small 3D models by twisting your arm (sometimes into an unnatural position), but as soon as you stop holding the trigger on the controller, the 3D model will fall to the floor.

In other words, all this is just not good for inspectability.

[elalish]

I do see what you mean, and I appreciate your perspective, but honestly the furniture use case is at least grounded in a somewhat proven business / user need. And I feel like the idea of an object dropping to the ground when it is let go of makes some intuitive sense, vs floating forever. Both modes would probably be nice, but complexity and discoverability are serious issues - for now trigger down vs trigger up feels like a pretty lightweight mode switch.

One thing - I believe we should be able to drop onto the local surface (e.g. table) instead of the original floor. That would help a bit at least. Any ideas on the best way to accomplish that?

[elalish]

One other question (@cabanier maybe you know?), since there's no DOM overlay in these headsets, is there a recommendation on how to exit a WebXR experience? I've seen ugly floating rectangles used for this, but that's about it. And when I use the home button and click "quit" on the background WebXR experience, I've been seeing some crashes - does that actually call the webxr session end event, or is it more of a force-quit?

[diegoteran]

Just a few nits for future readability.

[elalish]

Okay, I added scaling and in-place rotation. Could use some polish and UXR, but this is probably mergeable now.

[FluorescentHallucinogen]

@elalish

UX design, research, or just musings/examples would all help

For inspiration, try Layout, Meta's new mixed reality app:

https://www.meta.com/experiences/layout/9298251876913852/

https://youtu.be/vJKuy6NT0JQ

[elalish]

For inspiration, try Layout, Meta's new mixed reality app:

Thanks, though I should probably write another blog post about 3D UX, this time in headsets. Part of what irritates me about headsets is that so much of their 3D UX is like what you show above: despite having 6DOF inputs, we're still using a bunch of 1D and 2D widgets as though we're still using a mouse, but now with cerebral palsy.

I'm attempting to point the way toward a 3D UX that actually takes advantage of 6DOF inputs and doesn't require such precise pointing or so many modes and widgets. I had hoped someone had done this before me, but maybe not.

[elalish]

Did some more testing, found and fixed a few edges cases. I think this is good to go now.

[diegoteran]

Lgtm.

Only small nit for future readability for people working on these files, every time there is a check for if (this.xrMode !== 'screen-space'), it could be a good idea to add a comment saying what we are doing in general that is not required on a phone.

[FluorescentHallucinogen]

@elalish See also Figmin XR for some UX ideas:

https://www.meta.com/ru-ru/experiences/figmin-xr-mixed-reality/6849182851823457/

I like that the user can bring closer and further away (this is something that is very much lacking in the current implementation!), scale and rotate objects using only one single controller.

See this video (the left part of screen at 20 second time): https://youtu.be/bXpQz_lR9lA?t=20

This can be very useful in terms of accessibility (for those people missing one arm). And for the new Logitech MX Ink Mixed Reality Stylus (3D pen) that is already supported in WebXR by Meta Quest Browser. See https://x.com/rcabanier/status/1841896955091550612. 😉