ETH Zürich: Manipulating virtual objects in an augmented reality environment ETH Zürich created a visuo-haptic augmented reality system, in which virtual objects can be seen and manipulated in a filmed real world environment. Therefore researchers at the Computer Vision Lab built a set-up including a position tracking device, a head-mounted camera see-through display and a camera mounted marker, which cover the visual part of the interaction. For the haptic feedback, a virtual probe has been integrated into the set-up, by which the user explores the objects. Click for Video! Object manipulation in AR set-up A specific challenge was to align the haptic and the tracker coordinate systems for the manipulation. To this end a new calibration method has been developed, which allows high fidelity haptic interactions with virtual and real objects over the same tool. To provide stable head tracking, visual landmarks were fixed in the environment. These pre-calibrated landmarks optimize the camera pose reported from the tracker. Various objects of different size, stiffness and texture have been manipulated within the set-up, ranging from a visuo-haptic AR ping-pong over poking a deformable cylinder (see video) to medical simulations.   Detailed information on set-up and implementation: The setup comprises an optical position tracking device OPTOTRAK 3020 manufactured by Northern Digital Inc., a head-mounted FireWire camera and a camera-mounted marker (see figure below). The camera and the marker are rigidly attached, therefore the camera-marker transformation is fixed and estimated by an offline calibration process. Given this transformation and the marker pose, the AR system can estimate the camera pose with respect to the tracker coordinate frame. OPTOTRAK 3020 (left) - Head mounted display (right) In a second step, a haptic device has been successfully integrated into this setup. To this end a new calibration method to exactly align the haptic and tracker coordinate system was developed, which allows high fidelity haptic interactions with virtual and real objects over the same tool. The figure below shows a set of 3D-correspondances between tracker and haptic coordinates before and after the calibration step. Correspondences before and after calibration In order to provide enough performance for the simulations, the framework has been distributed and a synchronized data exchange mechanism, which ensures a low latency, has been developed. In the current system two machines are used – a graphics and a physics server. While the former carries out all vision and augmented reality specific tasks, the latter controls the haptic device and sends updates for visual rendering of the models to the graphics machine. This allows rendering of haptic feedback without being affected by the visual modality and vice versa. The data exchange is illustrated in the figure below. Furthermore, an extension of the distributed system allows integration of additional haptic devices via additional machines. To further improve synchronization and stability of the framework a hardware trigger signal has been integrated, which minimizes the temporal offsets between tracking and image data by hardware.            Communication model to exchange visual data Furthermore, to provide stable head tracking, a landmark based camera pose refinement was integrated. The approach uses visual pre-calibrated landmarks (see figure below) to optimize the camera pose reported from the tracker. In order to allow interaction in the workspace without affecting the stability, a visibility test to discard occluded landmark has been integrated. With this we could demonstrate that even when landmarks were temporarily occluded and the camera was moving a stable virtual overlay can be achieved. The figure below depicts the backprojection error before and after visual correction in relation to the number of visible landmarks of two centred landmarks. a) Artificial landmarks used by the vision-based tracking system; b) Backprojection error of two centered landmarks. During the development of the system several applications have been developed to test the fidelity and capabilities of the system. One of them is illustrated in the video above.

Related deliverables:

Related papers:

>>back to overview