Mode of Operation
Creating real and virtual worlds
Furnishing the real world
A trip to IKEA makes it possible. The living-room looks no longer like a laboratory. Since there are no more patterns needed for optical spatial tracking, the walls can be covered as well.
Reconstructing a virtual version of the real world
The entire living room including all furniture needs to be available as an accurate 3D model to allow realistic rendering of virtual objects (for example virtual objects casting shadows on real objects, real objects occluding virtual objects etc.). This was done in Cinema4D and 3D Studio Max.
First scenarios created with a simulator
The modeled living room was imported into Virtools and allowed us to create first simulations of scenarios (pure VR-scenarios with a virtual avatar, controllable with keyboard and mouse). The pictures show an external view of the avatar in the living room and the "first-person" view of the avatar.
Merging real and virtual worlds
Basic stereoscopic camera see-through setup and calibration
For stereoscopic viewing in the Virtools VR Player, a single virtual camera is modeled as two cameras with identical focal lengths and orientation. Each camera is horizontally displaced to the left or right of the single virtual camera by half the interpupillar distance (IPD). The real-world cameras are set up and mounted to the HMD in the same manner. Due to the casing of the cameras used, the distance between the real cameras could not be less than 75mm, even when the cameras were mounted with no gap between the casings. The IPD was thus set to the same value of 75mm, which is slightly more than usual IPD values (63mm is a widely used standard value). Nevertheless, the stereoscopic effect was tested in a VR-scenario and proved to be quite satisfactory, although slightly exaggerated.
In principle, the basic real-world view (we call this the AR backdrop) can now be obtained by displaying the image of the left camera on the left eye's display and the right camera on the right eye. However, the unprocessed camera images to not correspond to an exact stereoscopic image due to mechanical tolerances and unavoidable imprecisions in the mounting of the cameras. Therefore a more sophisticated setup on the software side is needed to obtain a satisfactory stereoscopic image.
Left: Reference pattern for stereo calibration.
Putting real and virtual views together
In the Virtools setup, the input from the head tracker is mapped to the position and orientation of a 3D coordinate system. The offset between the head-tracker and the camera is measured (on the real HMD) and transferred to a fixed offset between the tracker reference coordinate system and the virtual camera.
This tracker/camera/backdrop system can be placed into the simulator scenarios to achieve a superimposition of virtual and real worlds. To fine-tune the overlap of virtual and real objects, the virtual models can be rendered with a wire-frame outline.
Eventually the models of the real-world objects are no longer rendered, leaving an image composed of the real-world backdrop and virtual objects. The models of real-world objects can still occlude virtual objects (if a virtual object is placed behind a real object, the AR backdrop will appear "in front" of the virtual object within the outline of the real object).
To achieve the best possible congruence between real and virtual worlds, an additional calibration procedure was undertaken: The HMD was placed on a wooden pedestal in the center of the living room facing the rear wall. The position of the tracker and the orientation of the HMD were measured as accurately as possible by hand using rulers and leveled using a spirit-level. Reference points were marked on the rear wall, and also were added to the 3D model. Minor adjustments were now made to make the virtual model match the real image as well as possible.