Added camera feed and plane detection (both vertical and horizontal)

[elmajime]

Camera feed is functional if using branch https://github.com/elmajime/godot/tree/camera_from_external_feed for Godot. A Merge request have been created on Godot's repository.
The link to the MR is available in the proposal godotengine/godot-proposals#10678.

This adds camera feed display and plane detection.

Code is present to handle light and depth estimation too but the later needs a new feature in Godot.
The light estimation doesn't work quiet yet.

Remaining will be :

• shrinking planes when needed (right now they just grow)
• correct handling of phone orientation (hard coded as being in landscape mode)
• implement depth estimation use in Godot's repository and reactivate the dedicated code in this one
• correct light estimation
• implement point cloud display
• implement image detection
• make a nicer demo

[elmajime]

@BastiaanOlij should we merge this branch to allow continuing the integreation of other features like image detection and such ?

[BastiaanOlij]

@BastiaanOlij should we merge this branch to allow continuing the integreation of other features like image detection and such ?

I'm happy for you guys to merge this if you, @paddy-exe and @LucaJunge are happy with it. We can further tweak things as David Snopek and you can all start polishing things up independently in this repo.

Do make sure that the CI issues are cleaned up first.

[LucaJunge]

Yes, that's good progress. I'm fine with merging, but would suggest excluding the .bak files from the commit and rather comment out the parts in the original file

[elmajime]

Yes, that's good progress. I'm fine with merging, but would suggest excluding the .bak files from the commit and rather comment out the parts in the original file

@BastiaanOlij This is due to godot-cpp not having the right extension_api file. What shall I do about it? Create a branch in godot-cpp with the new file?

[BastiaanOlij]

Yes, that's good progress. I'm fine with merging, but would suggest excluding the .bak files from the commit and rather comment out the parts in the original file

@BastiaanOlij This is due to godot-cpp not having the right extension_api file. What shall I do about it? Create a branch in godot-cpp with the new file?

I think I answered this on chat but best to record it here as well.

What we do in other plugins is extract a fresh extension_api.json file from a custom build of Godot and place that inside of the repo. Then update the build instructions to use that instead of the default one.

[paddy-exe]

Looks good to me. Can be merged.

[BastiaanOlij]

Happy to merge this is we can get CI to finish.
Most important thing is that when you compile with a custom extension json file, that file should live in the main repo so people have access to it.

Also on a more general note, I think long term we should make plane detection work similarly to how we've solved this originally in ARKit (in Godot 3) and for Metas scene discovery.
E.g. each detected plane registers and Anchor on the XRServer and this drives what is added/created so the user can deviate from the base logic.
That said, I don't think this should hold things back, we can always deprecate the current approach and it may be better timing once we know what OpenXR will come up with as a standard here.

[elmajime]

@BastiaanOlij is it possible to run the CI? I think I've corrected the problem (the path in the command was wrong) But i'ld like to be sure.

[BastiaanOlij]

@BastiaanOlij is it possible to run the CI? I think I've corrected the problem (the path in the command was wrong) But i'ld like to be sure.

Started it back up again, I'm not sure when GitHub decides to run these automatically, I think you need atleast one merged PR in the repo, so feel free to keep pinging me.

That said, if this passes I'm all for merging, we can fix further problems with new PRs.

[elmajime]

Happy to merge this is we can get CI to finish. Most important thing is that when you compile with a custom extension json file, that file should live in the main repo so people have access to it.

Also on a more general note, I think long term we should make plane detection work similarly to how we've solved this originally in ARKit (in Godot 3) and for Metas scene discovery. E.g. each detected plane registers and Anchor on the XRServer and this drives what is added/created so the user can deviate from the base logic. That said, I don't think this should hold things back, we can always deprecate the current approach and it may be better timing once we know what OpenXR will come up with as a standard here.

Hey ! I'ld like to get this working, @BastiaanOlij where could we start a conversation about this? I've looked at the code for the ARKit version, I like the idea though I don't understand how the bounds of the planes are held by the anchors.