We synthesise the views for the left and right eyes by projecting the camera image onto some assumed world geometry. Our 6-DoF experiences are generally designed for a table-top setting, and we project the camera onto an infinite plane on that table surface (there's a physical origin marker to define the origin of the space for the 6-DoF experiences). There's also a plane parallel to the device a few meters in the distance (to avoid a "horizon").

Sounds simple but for table-top experiences like viewing 3D models on a table it does give a convincing stereo effect - things like shadows from the model on the table just look right. Combined with the really low-latency display pipeline[1], the nice camera (1440x1080p @ 60fps), and the direct peripheral view of the real world thanks to the headset design gives a really pretty comfortable MR experience.

The next job will be improving the quality of the world geometry so that camera reprojection works better for static parts of the scene. We might try something live with lidar but for most rooms an offline process (or even manual placement and scaling of planes for walls / floors) would already provide a decent improvement.

The hardest part is foreground objects - hands, controllers, other people. Some scheme to detect and segment out those foreground bits and reproject them separately using an estimated depth map (whilst infilling the background layer for the static world) would be needed there. Complex in general but for things with known geometry ie controllers, it seems like something we should be able to do.

What's hopefully clear is that we're pushing to make the headset MR experience using your phone the best it can possibly be, given the limitations imposed by the hardware such as camera positioning. We're in a pretty good place already but there are definitely more improvements to come :)

[1] As this is HN, I can say a bit more technical stuff about the display pipeline... We have an iOS implementation for async timewarp - we've been able to circumvent the usual queuing of frames that happens on iOS to implement a late warp render just 3ms before a new frame starts scanning out. That means we can get a really recent camera frame (on iOS frames are delivered to the app around 20ms after capture) and also apply rotational reprojection for any movement since the frame was captured. Reprojecting for predicted translation at display time is on the roadmap too which should remove the last bit of noticeable latency when moving.

We've definitely had the same experience of Safari introducing way more issues with new releases than Chrome, and the fact it's part of the iOS image seems to make quick patching just never really an option.

I just wish they'd engage a bit more directly on the bug tracker - find it hard to believe that twitter is a more effective means to get attention on bugs.

I've tried on some of the glasses from Meta and Epson that consist of more traditional micro-displays and optics to set a fixed virtual focus plane a couple of metres away. It's definitely possible to shift attention between the real and the virtual, and not feel too distracted even with relatively busy background scenes (using some of the "shades" definitely helps to boost contrast of the virtual content though). When real and virtual are at different focal planes it seemed to make maintaining attention on the virtual content easier, and stereo also seems to help with that.

However the true promise of "Mixed Reality" experiences are when you don't have to choose to attend to real or virtual, but simply look at a part of the combined scene. When virtual and real are so closely connection it seems likely to me that the depth of the virtual content would need to be matched with to the depth of the real world, so the virtual and real focal planes would be the same and the user would lose the ability to use depth cues to direct attention.

If real and virtual are at the same depth, how would you imagine the scene would look? Is the additive mock-up still unrealistic?