I’m curious if anyone has developed an AR app that looks like this concept video I just watched. It seems cool and has so much potential. What do you all think?
Credit: Motion Design by Pavel Khomin
This looks like it would be tough to create. It probably needs machine learning for recognizing buildings and then procedural generation for the shapes and textures. An easier way might be to use a world tracker or a VPS to display a 3D model in a specific building.
I think sky segmentation is already a thing, and 3D building geometry will be available soon in AR toolkits, right?
I don’t think sky segmentation would work well if the goal is to replace a building. Plus, making this for AR might be tricky unless we have AR glasses. Using a phone could be a hassle.
What if we segment the sky and fill in the gaps behind a building? If there’s another building behind the one we want to replace, we could pull a texture from a database. I saw a demo using the Project PLATEAU dataset for Tokyo. We could use textures from there for the Tetris building and adjust them based on lighting. I guess we’d need machine learning to break the textures into bricks that fit. It doesn’t have to be in real-time, right? Users could download a texture pack.
About mobile vs AR/MR, that’s a good point. With a phone, everything would be on the same depth plane. But with optical see-through, it gets complicated.
Yeah, that could work for some central buildings.
It would be easier if the building didn’t move. Still pretty cool though.
Maybe you’re right. Using the world as a canvas sounds nice. I remember a game from the ARCore Geospatial API announcement where they used a building as a background for a platformer game. If it makes sense and isn’t just for show, changing the world around us would be even cooler.
Imagine if you could look at a building and see inside to view its inhabitants. You could even zoom in for details. That would be hilarious! Or seeing people’s skeletons would be wild, but that might be harder to create.
Oh wow! That’s a fantastic idea! You could walk through the city to unlock story pieces that play inside buildings. Maybe something like Wall Town Wonders but in the real world. 
We should keep it random, though, so all players don’t end up at the same buildings and annoy the locals. Plus, there might be property rights to think about if we display ads on certain buildings.
Dat seamless loop dough.
“I told you… the building is closing NOW! GET OUT!”
It’s going to be mind-blowing when machine learning can recognize the world around us and blend AR into it. It’ll be a trippy experience.
Right? Even doing laundry will feel trippy.
It could be simple in the sky if the whole building is virtual.
Yes, that’s easier. But maybe not as exciting. Players might not want to share videos if everything looks the same.
For sure. Bringing down a real building like that would look way more satisfying!