Modeling free-hand object manipulation (P2O) |
|
|
From observations of how people manipulate real objects with their hands, typical hand poses and the corresponding multimodal sensations are investigated. This not only allows us to replay such manipulations virtually, but also mimic manipulation experiences for new objects. Several components will be integrated to achieve these goals. A first module will observe how it looks and feels when an object is manipulated. This module will consist of two main components. On the one hand a tactile haptic sensor glove will be used, to measure forces, positions, temperatures, and surface roughness at multiple contact points throughout the manipulations. Simultaneously, the ‘in-hand scanning’ 3D acquisition system will record the events in 3D. Dense range measurements will be made of the hands and the object. This will allow the system to not only track the hands in 3D over time, but at the same time also build a 3D object model. Together, these components will form a dynamic touch-print of the hands in relation to the object. Detailed haptic data and hand poses in relation to a 3D object model will result. This first functionality is what we refer to as ‘record’. A second goal is then to regenerate the real experience as realistically as possible, using a haptic display and graphics-based visualization. The haptic display yields a haptic experience, while the image-based rendering techniques yield photo-realistic visual renderings of the object as it would be seen under the relevant poses. In order to complete the experience, a rendering of virtual hands will be added, with additional shadowing effects between object and hands. Although challenging, this functionality is limited to a literal ‘replay’. Yet, it already supports important... |
... applications like virtual hands-on training, for instance an inexperienced trainee quite literally getting into the skin of a world-renowned surgeon and following and feeling her actions. The functionality will in a next step be extended, in order to support experiences in more versatile virtual and mixed worlds. Through the analysis of touch-prints from former experiments, the system will generalize towards the handling of objects for which no such recordings have been made and / or where the user himself is in control of the motions. It will also be possible to enhance the user's cognitive functions by systematically comparing incoming data with those stored in the database (e.g. for medical diagnosis through palpation of a real patient, it will be possible to compare against data from the patient several weeks earlier) and to ‘augment’ haptic sensations to highlight interesting features (e.g. providing a mode where in the previous example the physician gets an additional, salient haptic pattern that indicates differences). In the first stage, the work will concentrate on the handling of rigid object. This will allow developing and testing the most basic technological components within an environment, where realistic haptic interaction between the hand and the object can be handled under significantly simplified conditions. Once these basic technologies are functional, the more challenging problem of handling soft (deformable) objects will be addressed. The real-time deformation during hand-object interaction will result in major challenges in all three stages of the modelling pipeline, but will ultimately allow dealing with a much broader spectrum of potential applications. |