Hey everyone, I’ve been working on this AR project that uses multiple deep learning models. I managed to get intrinsics and extrinsics (cam2world matrices) along with depth images from various frames of a video. So far, I’ve been able to render a 3D object that looks like it’s in the scene when it was captured. The problem is that the object seems to be floating instead of being pinned to something in the scene. I realize now that I need to use the depth maps to anchor it properly. Any advice on how to move forward would be super helpful.
For the best results, you’ll need to run a robust Visual Inertial SLAM combined with a low poly point cloud. That should help with proper anchoring.
We did something similar using ML to place AR content. We found that providing labeled data with bounding boxes in 3D helped a lot. It was all positioned relative to the object we wanted to anchor to.
IDK if this helps, but have you thought about adjusting the depth maps to ensure they align with the real-world surfaces? That could be key to anchoring your object.
Have you considered using point cloud data along with depth images? It might give you a clearer idea of where to position your 3D object.