BACKGROUND: While an update rate of 30 Hz is considered adequate for real time graphics, a much higher update rate of about 1 kHz is necessary for haptics. Physics-based modeling of deformable objects, especially when large nonlinear deformations and complex nonlinear material properties are involved, at these very high rates is one of the most challenging tasks in the development of real time simulation systems. While some specialized solutions exist, there is no general solution for arbitrary nonlinearities. METHODS: In this work we present PhyNNeSS - a Physics-driven Neural Networks-based Simulation System - to address this long-standing technical challenge. The first step is an off-line pre-computation step in which a database is generated by applying carefully prescribed displacements to each node of the finite element models of the deformable objects. In the next step, the data is condensed into a set of coefficients describing neurons of a Radial Basis Function network (RBFN). During real-time computation, these neural networks are used to reconstruct the deformation fields as well as the interaction forces. RESULTS: We present realistic simulation examples from interactive surgical simulation with real time force feedback. As an example, we have developed a deformable human stomach model and a Penrose-drain model used in the Fundamentals of Laparoscopic Surgery (FLS) training tool box. CONCLUSIONS: A unique computational modeling system has been developed that is capable of simulating the response of nonlinear deformable objects in real time. The method distinguishes itself from previous efforts in that a systematic physics-based pre-computational step allows training of neural networks which may be used in real time simulations. We show, through careful error analysis, that the scheme is scalable, with the accuracy being controlled by the number of neurons used in the simulation. PhyNNeSS has been integrated into SoFMIS (Software Framework for Multimodal Interactive Simulation) for general use.

What are desirable and undesirable features of virtual-environment (VE) software architectures? What should be present (and absent) from such systems if they are to be optimally useful? How should they be structured? To help answer these questions we present experience from application designers, toolkit designers, and VE system architects along with examples of useful features from existing systems. Topics are organized under the major headings of: 3D space management, supporting display hardware, interaction, event management, time management, computation, portability, and the observation that less can be better. Lessons learned are presented as discussion of the issues, field experiences, nuggets of knowledge, and case studies.

Surgical simulators present a safe and potentially effective method for surgical training, and can also be used in robot-assisted surgery for pre- and intra-operative planning. Accurate modeling of the interaction between surgical instruments and organs has been recognized as a key requirement in the development of high-fidelity surgical simulators. Researchers have attempted to model tool-tissue interactions in a wide variety of ways, which can be broadly classified as (1) linear elasticity-based, (2) nonlinear (hyperelastic) elasticity-based finite element (FE) methods, and (3) other techniques that not based on FE methods or continuum mechanics. Realistic modeling of organ deformation requires populating the model with real tissue data (which are difficult to acquire in vivo) and simulating organ response in real time (which is computationally expensive). Further, it is challenging to account for connective tissue supporting the organ, friction, and topological changes resulting from tool-tissue interactions during invasive surgical procedures. Overcoming such obstacles will not only help us to model tool-tissue interactions in real time, but also enable realistic force feedback to the user during surgical simulation. This review paper classifies the existing research on tool-tissue interactions for surgical simulators specifically based on the modeling techniques employed and the kind of surgical operation being simulated, in order to inform and motivate future research on improved tool-tissue interaction models.

Image quality issues such as field of view (FOV) and resolution are important for evaluating "presence" and simulator sickness (SS) in virtual environments (VEs). This research examined effects on postural stability of varying FOV, image resolution, and scene content in an immersive visual display. Two different scenes (a photograph of a fountain and a simple radial pattern) at two different resolutions were tested using six FOVs (30, 60, 90, 120, 150, and 180 deg.). Both postural stability, recorded by force plates, and subjective difficulty ratings varied as a function of FOV, scene content, and image resolution. Subjects exhibited more balance disturbance and reported more difficulty in maintaining posture in the wide-FOV, high-resolution, and natural scene conditions.

Two experiments examined perceived spatial orientation in a small environment as a function of experiencing that environment under three conditions: real-world, desktop-display (DD), and head-mounted display (HMD). Across the three conditions, participants acquired two targets located on a perimeter surrounding them, and attempted to remember the relative locations of the targets. Subsequently, participants were tested on how accurately and consistently they could point in the remembered direction of a previously seen target. Results showed that participants were significantly more consistent in the real-world and HMD conditions than in the DD condition. Further, it is shown that the advantages observed in the HMD and real-world conditions were not simply due to nonspatial response strategies. These results suggest that the additional idiothetic information afforded in the real-world and HMD conditions is useful for orientation purposes in our presented task domain. Our results are relevant to interface design issues concerning tasks that require spatial search, navigation, and visualization.

Observers adjusted a pointer to match the depicted distance of a monocular virtual object viewed in a see-through, head-mounted display. Distance information was available through motion parallax produced as the observers rocked side to side. The apparent stability of the virtual object was impaired by a time delay between the observers' head motions and the corresponding change in the object position on the display. Localizations were made for four time delays (31 ms, 64 ms, 131 ms, and 197 ms) and three depicted distances (75 cm, 95 cm, and 113 cm). The errors in localizations increased systematically with time delay and depicted distance. A model of the results shows that the judgment error and lateral projected position of the virtual object are each linearly related to time delay.

The question of whether the sense of presence in virtual environments (or telepresence with respect to teleoperator systems) is causally related to task performance remains un-answered because the appropriate studies have yet to be carried out. In this brief report, the author describes a strategy for resolving this issue and the results of a pilot study in which this strategy was implemented.

For entertainment applications, a successful virtual experience based on a head-mounted display (HMD) needs to overcome some or all of the following problems: entering a virtual world is a jarring experience, people do not naturally turn their heads or talk to each other while wearing an HMD, putting on the equipment is hard, and people do not realize when the experience is over. In the Electric Garden at SIGGRAPH 97, we presented the Mad Hatter's Tea Party, a shared virtual environment experienced by more than 1,500 SIGGRAPH attendees. We addressed these HMD-related problems with a combination of back story, see-through HMDs, virtual characters, continuity of real and virtual objects, and the layout of the physical and virtual environments.

Ten subjects adjusted a real-object probe to match the distance of nearby virtual objects optically presented via a see-through, helmet-mounted display. Monocular, binocular, and stereoscopic viewing conditions were used with two levels of required focus. Observed errors may be related to changes in the subjects' binocular convergence. The results suggest ways in which virtual objects may be presented with improved spatial fidelity.

Virtual reality technology is now being used to provide exposure and desensitization for a number of phobic conditions. In this paper, we first review these current applications and discuss the work needed to refine and expand these applications to phobias. We then comment briefly on some preliminary applications of VR technology to mental-health problems outside the domain of phobias. Finally, we consider ways in which VR might be used to further enhance psychotherapy and assist in the treatment of a wide variety of disorders. Various possible interventions are discussed, along with the technological developments needed to make them possible.

The sensory and motor capacities of the human hand are reviewed in the context of providing a set of performance characteristics against which prosthetic and dextrous robot hands can be evaluated. The sensors involved in processing tactile, thermal, and proprioceptive (force and movement) information are described, together with details on their spatial densities, sensitivity, and resolution. The wealth of data on the human hand's sensory capacities is not matched by an equivalent database on motor performance. Attempts at quantifying manual dexterity have met with formidable technological difficulties due to the conditions under which many highly trained manual skills are performed. Limitations in technology have affected not only the quantifying of human manual performance but also the development of prosthetic and robotic hands. Most prosthetic hands in use at present are simple grasping devices, and imparting a "natural" sense of touch to these hands remains a challenge. Several dextrous robot hands exist as research tools and even though some of these systems can outperform their human counterparts in the motor domain, they are still very limited as sensory processing systems. It is in this latter area that information from studies of human grasping and processing of object information may make the greatest contribution.

Model-based approaches can be used to confront several of the challenging performance issues in teleoperation. This paper describes a model-based supervisory control technique for telerobotics. A human-machine interface (HMI) was developed for online, interactive task segmentation and planning utilizing a world model of the telerobotic working environment (TRWE). The task model is transferred intermittently over a low bandwidth communication channel for interpretation, planning, and execution of the task segments through the autonomous control capabilities of a telerobot. For the purposes of outlining tasks, a human operator controls a simulation model to generate a "task sequence script" as a sequential list of desired sub-goals for a telerobot. A graphic user interface (GUI) facilitates the development of the task sequence script with viewing perspectives of the graphic display automatically selected as a function of the operational state and model parameters. Also, because the human operator is specifying discrete model set-points of the TRWE, and allowing the autonomous control capabilities of the telerobot to coordinate the actual trajectory between set-points, a provision is made to preview the proposed trajectory for approval or modification before execution. Preliminary results with a manipulator arm remotely controlled via the Internet demonstrate the utility of the model-based supervisory control technique.

An operator's sense of remote presence during teleoperation or use of virtual environment interfaces is analyzed as to what characteristics it should have to qualify it as an explanatory scientific construct. But the implicit goal of designing virtual environment interfaces to maximize presence is itself questioned in a second section in which examples of human-machine interfaces beneficially designed to avoid a strong sense of egocentric presence are cited. In conclusion, it is argued that the design of a teleoperation or virtual environment system should generally focus on the efficient communication of causal interaction. In this view the sense of presence, that is of actually being at the simulated or remote workplace, is an epiphenomena of secondary importance for design.

Overall system latency--the elapsed time from input human motion until the immediate consequences of that input are available in the display--is one of the most frequently cited shortcoming of current virtual environment (VE) technology. Given that spatial displacement trackers are employed to monitor head and hand position and orientation in many VE applications, the dynamic response intrinsic to these devices is an unavoidable contributor to overall system latency. In this paper, we describe a testbed and method for measurement of tracker dynamic response that use a motorized rotary swing arm to sinusoidally displace the VE sensor at a number of frequencies spanning the bandwidth of volitional human movement. During the tests, actual swing arm angle and VE sensor reports are collected and time stamped. By calibrating the time stamping technique, the tracker's internal transduction and processing time are separated from data transfer and host computer software execution latencies. We have used this test-bed to examine several VE sensors--most recently to compare latency, gain, and noise characteristics of two commercially available electromagnetic trackers: Ascension Technology Corp.'s Flock of Birds(TM) and Polhemus Inc.'s Fastrak(TM).

We present a model for simulating casualties in virtual environments for real-time medical training. It allows a user to choose diagnostic and therapeutic actions to carry out on a simulated casualty who will manifest appropriate physiological, behavioral, and physical responses. Currently, the user or a "stealth instructor" can specify one or more injuries that the casualty has sustained. The model responds by continuously determining the state of the casualty, responding appropriately to medical assessment and treatment procedures. So far, we have modeled four medical conditions and over 20 procedures. The model has been designed to handle the addition of other injuries and medical procedures.

This paper presents a body model server (BMS) that provides real-time access to the position and posture of a person's torso, arms, hands, head, and eyes. It can be accessed by clients over a network. The BMS is designed to function as a device-independent data-layer between the sensing devices and client applications that require real-time human motion data, such as animation control. It can provide clients with accurate information at up to 40 Hz. For data collection, the model uses four magnetic position/orientation sensors, two data-gloves, and an eye-tracker. The BMS combines the data-streams from the sensors and transforms them into snapshots of the user's upper-body pose. A geometric model made up of joints and segments structures the input. Posture of the body is represented by joint angles. Two unique characteristics of our approach are the use of the implicit, geometric constraints of the sensed body to simplify the computation of the unmeasured joint angles, and the use of time-stamped data that allow synchronization with other data streams, e.g., speech input. This paper describes the architecture of the BMS, including the management of multiple input devices, the representation and computation of the position and joint angle data, and the client-server interface.

A simple inverse kinematics procedure is proposed for a seven degree of freedom model of the human arm. Two schemes are used to provide an additional constraint leading to closed-form analytical equations with an upper bound of two or four solutions. Multiple solutions can be evaluated on the basis of their proximity from the rest angles or the previous configuration of the arm. Empirical results demonstrate that the procedure is well suited for real-time applications.

Simulating a human figure performing a manual task requires that the agent interact with objects in the environment in a realistic manner. Graphic or programming interfaces to control human figure animation, however, do not allow the animator to instruct the system with concise "high-level" commands. Instructions coming from a high-level planner cannot be directly given to a synthetic agent because they do not specify such details as which end-effector to use or where on the object to grasp. Because current animation systems require joint angle displacement descriptions of motion--even for motions that incorporate upwards of 15 joints--an efficient connection between high-level specifications and low-level hand joint motion is required. In this paper we describe a system that directs task-level, general-purpose, object grasping for a simulated human agent. The Object-Specific Reasoner (OSR) is a reasoning module that uses knowledge of the object of the underspecified action to generate values for missing parameters. The Grasp Behavior manages simultaneous motions of the joints in the hand, wrist, and arm, and provides a programmer with a high-level description of the desired action. When composed hierarchically, the OSR and the Grasp behavior interpret task-level commands and direct specific motions to the animation system. These modules are implemented as part of the Jack system at the University of Pennsylvania.

This paper presents a unique virtual reality training and assessment tool developed under a NASA grant, "Research in Human Factors Aspects of Enhanced Virtual Environments for Extravehicular Activity (EVA) Training and Simulation." The Remote Access Virtual Environment Network (RAVEN) was created to train and evaluate the verbal, mental and physical coordination required between the intravehicular (IVA) astronaut operating the Remote Manipulator System (RMS) arm and the EVA astronaut standing in foot restraints on the end of the RMS. The RAVEN system currently allows the EVA astronaut to approach the Hubble Space Telescope (HST) under control of the IVA astronaut and grasp, remove, and replace the Wide Field Planetary Camera drawer from its location in the HST. Two viewpoints, one stereoscopic and one monoscopic, were created all linked by Ethernet, that provided the two trainees with the appropriate training environments.

In the teleprogramming system an operator is presented with a virtual reality representation of a remote environment. The operator's interaction within that virtual environment is observed and translated into a sequence of robot program instructions for transmission to, and execution by, a robot at the remote site. In this paper we focus on operator interaction with the master station of the teleprogramming system. The use of synthetic fixtures to provide force and visual clues to assist the operator in performing tasks with speed and precision is discussed. It is suggested that, at least in some situations, it is both necessary and desirable to trade off realism for improved task performance. The difficulty of coping with exceptional conditions and, in particular, uncertainty in the world model used to generate the virtual environment is described and the operator interface for diagnosing and resolving errors is presented. An overview is also given of both the hardware and software used to implement master station for the teleprogramming system.

Given the ease that humans have with using a keyboard and mouse in typical, non-colocated computer interaction, many studies have investigated the value of co-locating the visual field and motor workspaces using immersive display modalities. Significant understanding has been gained by previous work comparing physical tasks against virtual tasks, visuo-motor co-location versus non-colocation, and even visuo-motor rotational misalignments in virtual environments (VEs). However, few studies have explored all of these paradigms in context with each other and it is difficult to perform inter-study comparisons because of the variation in tested motor tasks. Therefore, using a stereoscopic fish tank display setup, the goal for the current study was to characterize human performance of a 3D Fitts' point-to-point reaching task using a stylus-based haptic interface in the physical, co-located/non-colocated, and rotated VE visualization conditions.Five performance measures - throughput, initial movement error, corrective movements, and peak velocity - were measured and used to evaluate task performance. These measures were studied in 22 subjects (11 male, 11 female, ages 20-32) performing a 3D variant of Fitts' serial task under 10 task conditions: physical, co-located VE, non-colocated VE, and rotated VEs from 45-315° in 45° increments.