living-room

Research Process
The research process took place simultaneously on different levels. While we evaluated different hardware and software configurations we developed different scenes to exemplify the possibilities of perception and interactivity with living-room. The first six months were dominated by technical aspects and questions which occurred while developing the technical setup. Only during the last 3 Months was it possible to design and test different scenarios. Here we limited the amount to four and did further system adjusting and implemented an automated scene recognition, triggered by pages of a book.

1. Technical process
As already mentioned, the development of the technical setup on the levels of soft- and hardware was very time consuming and opened a large field of investigation. Firstly, because almost all known AR-systems are object-oriented and do not use rooms as tracking systems and secondly because of the high expectations we have towards a large design-vocabulary and which had to be easy to handle.

1.1 Building the living-room
The living-room was designed by the following criteria:
The user was to have the minimum amount of space in which he or she can move and interact without having the feeling of being too constricted or confined.
Spectators should be able to simultaneously follow the movements of the user in real-space from outside and be able to watch the augmented pictures on a computer screen.
The construction was carried out in light and cost-efficient but durable cardboard elements to enable an assembly within two hours. This and the attractive design were an important issue for the later use at different exhibitions and festivals.

1.2 Evaluation of the software configuration
Different software combinations under Mac OS X have been evaluated and tested.
The evaluation of the software configuration had to meet the following requirements:
Inexpensive, compatibility, connectivity, user friendliness, performance, exact tracking, no limitations to interactivity and design possibilities. Furthermore our system was to be adaptable to the future introduction of various sensors to enhance interactivity.

1.2.1 Evaluation of tracking systems
ARToolKit combines the tracking of markers and augmenting objects on to the recognized markers. The relative position of the camera towards the marker is given to the 3D-engine which augments 3D-elements into the video stream. Our tests with ARToolKit resulted to following conclusions. ARToolKit could meet our demands concerning tracking with the disadvantage that living-room had to be wallpapered with markers. The absolute position of the camera was to be calculated for every marker. This also meant that the interior design in real space was restricted and tracking problems due to occlusion could occur. Tests made with infrared reflective markers to reduce the dominant appearance of the markers failed due to imprecise and poor recognition.
Our tests to augment objects with ARToolKit could not fulfill our expectations. Particularly due to the poor import, user interface and animation possibilities, further experiments were abandoned.
The question of marker-less tracking and tracking with radio waves were brought up in theory. During the research process connections to other academic institutes developing such applications have been made and a collaboration is planned in the future.

1.2.2 Evaluation of AR Software
The possibility to develop an interface from ARToolKit as tracking software and a third party software to augment virtual objects in a user friendly environment lead to numerous amounts of tests and consultations. General requirements for the AR software were the ability to import animated and textured 3D-models, to display them on a real time video stream, user friendliness, interface to ARToolKit and other sensor oriented applications and the possibility to program events due to the user's actions.

1.2.2.1 Macromedia Director
Macromedia director provides the well performing 3D interface "Shockwave 3D" and offers an interface to external software. Due to our experience in Lingo which allows event handling we were encouraged to experiment by augmenting 3D-animations over a realtime video stream. Director failed to place objects over moving images with useful frame rates.

1.2.2.2 Max/MSP/Jitter
Jitter provides a convenient and variable interface in which 3D-models can be super-imposed over video streams. Many design-oriented features (texturing, lighting, animation, sounds) are included with many well documented examples. Event handling and actions are easy to program. Real time camera positions and angles can be simply imported to control the 3D display. A weakness we encountered, was that only the most basic polygonal models could be imported. Jitter itself provides a limited amount of simple primitives but offers more and other modeling tools not found elsewhere. Nevertheless all other arguments were preponderate which lead us to the decision to continue with the combination of ARToolKit and Jitter.

DOWNLOAD: model2.pdf

1.2.2.3 Unreal Tournament engine
Inspired by speed, ambiance, crash detection and network collaboration of upcoming computer games we took a short look at the possibilities such engines would have in the future for augmented applications. We withdrew from the idea because of the difficult programming environment particularly in its combination with ARToolKit as the tracking software.

1.2.3 Programming the interface between ARToolKit and Max/MSP/Jitter
The interface was programed by I-Art Valentine Spiess in Basel CH. A detailed documentation can be requested from j.torpus@fhbb.ch

1.3 Evaluation of the hardware configuration
Different hardware configurations were considered and tested. The operating system was postulated to be Mac OS X as it is dominantly used at the University of Art and Design in Basel

1.3.1 Wearable configuration
The wearable configuration with a wearable 12" Macintosh G4 800 MHz Powerbook communicating via wireless lan with a G4 Desktop was dropped because of performance reasons. Nevertheless future wearable fast computers are a precondition to expanding the user's mobility and the dimension of augmented space.

1.3.2 Desktop configuration
The desktop configuration with one Macintosh dual processor G5 2.0 GHZ Desktop computer with two firewire cameras (tracker and viewer) and a head-mounted display directly connected, enabled the satisfactory performance of 25 frames per second and more. The drawbacks of this configuration are that the user is connected to the room by cables and thus reducing his actions to a 5m radius.

2. Designing Scenarios and system adjusting
more to come!

3. OPENING

January 23rd 2004 - Grand opening and presentation to the public
In addition to the living-room we also had a station with a living-room simulation (an adapted Max/MSP/Jitter patch in combination with the "little portable living room") to explain the scenarios and to give instructions for the visit of the physical installation.
The guests enjoyed the different scenarios and the feedback was generally very positiv. Click >here< to see a slideshow of the event.

4. Statements

>here< are some statements - some in German, some in English - from visitors to the installation.