The main computer is the heart of the system, where all inputs (cameras, tracking data, sensors) come together and are processed to generate the visual output as well as audio and signals for controlling the real-world devices. The computer used is a standard PC running Windows XP, with an Intel Core Duo 6600 CPU, 2 GB RAM and an Nvidia EN7900GTX graphics processor. This GPU provides enough power to render the AR scenarios in adequate quality and is equipped with two separate DVI outputs and can thus be used for stereoscopic rendering.
Visual system: head-mounted display (HMD) with camera see-through
To achieve the desired composition of a real-world image overlayed by virtual 3D-objects, we use a display system involving a high-resolution stereo HMD together with a stereoscopic camera setup. This provides the user with a basic, realistic view of the real world, while technically opening the possibility to manipulate the user's view.
The HMD used is the nVisor SX manufactured by NVIS. This HMD has two separately controllable LCOS microdisplays (one per eye), each with a field-of-view of 60° and a resolution of 1280x1024 (SXGA). This HMD has a hard plastic enclosure which allows relatively easy mounting of additional hardware, e.g. cameras and head-tracker. Headphones are also integrated in the casing.
After evaluating various low-cost webcams, we eventually decided to use higher-quality industrial cameras. In the final system, we used two Dragonfly2 cameras manufactured by Point Grey Research with 1/3" CCDs with VGA resolution, together with Fujinon vari-focal lenses with manual zoom, focus and iris. The selected camera/lens combination has several advantages over lower-cost webcams:
To allow monitoring of the user's view, the DVI outputs of the PC are split using active DVI splitters, allowing the image for each eye to be sent to a large LCD screen as well as to the HMD.
3D Software Environment / Rendering Engine
The 3D software environment Virtools was used to build the AR scenarios. This platform provides a sophisticated development environment for interactive 3D content as well as a high quality 3D rendering engine. In the visual programming environment, so-called "building blocks" providing various functionalities can be combined in a drag-and-drop manner and thus used to implement interactivity and manipulate 3D objects. The available building blocks allow complex interactive 3D scenarios to be implemented without low-level coding.
Functionalities which can not be directly implemented in the visual programming environment may be realized either using a built-in scripting engine or by extending the environment by programming plug-ins in C++ using the Virtools SDK.
To accurately superimpose virtually rendered objects onto the real-world camera image, the position and orientation of the cameras must be known and correspond to the position of the virtual camera.
This is achieved by attaching a head-tracking device to the HMD enclosure. The offset between the head-tracker and the cameras is assumed to be constant, so that the position and orientation of the cameras can be easily calculated from the measured position and orientation of the head-tracker. We use an Intersense IS-900 hybrid tracking system which combines ultrasonic time-of-flight measurements with inertial sensor data to provide high accuracy and tracking stability. This system consists of the following components:
Calibrating the IS-900 tracking system
For the tracking system to be able to provide accurate data, the exact position of each ultrasonic speaker (Sonipod) must be determined. This was achieved with the help of the "Institute for Measurement and Geo-Information, FHNW", who performed the measurements using professional equipment from Leica Geosystems.
With these measurements, the tracking system can be configured for our set-up. Using a customized VRPN-server the tracking data can be made available in the Virtools environment.
One goal of the project was to explore possibilities of interactivity between real-world and virtual objects. The following real-world objects are interfaced to the system:
The real-world interfaces are controlled using a Media Controller IPX by tegoro solutions ag. This is a microcontroller-based device with Ethernet, DMX, serial, analog and digital I/O connections. Custom firmware was developed together with a Virtools plug-in to implement an interface to the main AR engine. The lamps and ventilator are connected to a DMX dimmer pack which is controlled by the Media Controller according to signals received from the main engine. The sensors used in the drawers are Sharp GP2D120 infra-red range sensors which supply an analogue signal to the Media Controller. The analogue inputs are converted to digital values, filtered and sent to the main engine by the Media Controller firmware. A simple UDP/IP protocol is used for communication between the Media Controller and the main engine.
Additional system for recording user sessions
To be able to evaluate users' behavior in the AR scenarios, we wanted to be able to record videos showing the rendered AR view as well as the user's reactions. A separate system with a PC, webcam and DVI frame grabber was built in addition to the main AR system to be able to record these videos of the users' experiences. The rendered 3D scene was captured and combined with images from a camera directed into the room. This recording functionality was implemented using Max/MSP/Jitter.