virtual
Content
Reflex Movements for a Virtual Human: a Biology Inspired Approach
Mario Gutierrez, Frederic Vexo and Daniel Thalmann
Virtual Reality Laboratory (VRlab) Swiss Federal Institute of Technology Lausanne (EPFL) Lausanne Switzerland CH-1015 {mario.gutierrez,frederic.vexo,daniel.thalmann}@epfl.ch
Abstract. This paper presents the results of a method to produce autonomous animation of virtual humans. In particular, the proposed methodology is focused on the autonomous synthesis of non-voluntary gestures such as reflexes and subtle movements which provide a noticeable impression of realism and naturalness. The final goal of this technique is to produce virtual humans with a more spontaneous, non pre-programmed behaviour. For the moment, the technique is applied to the synthesis of reflex movements of the arm, in reaction to thermic stimuli. Nevertheless, a general architecture is outlined.
1
Introduction
The animation of virtual humans and computer animation in general have always searched to produce realistic imagery and true naturalness in the motions. One particularly difficult problem is to provide automatic animation of a virtual human displaying a behaviour that truly resembles a real human [1], [2]. Our goal is to animate virtual humans (body gestures) in an autonomous way, giving the illusion that the artificial human being is a living creature displaying spontaneous behaviour according to its internal information and external stimuli coming from its virtual environment. The proposed method is a real-time, distributed control system inspired in the human nervous system, which will be able to produce natural movements in response to external and internal stimuli. The advantage of the real-time control techniques over the ”Computer Graphics movie” approach is their promise of combined autonomy and realism as well as their focus on reusability. The autonomous control system we are presenting could be instantiated and used to animate several different characters in a variety of situations. This paper is organized according to the following structure: in the next section we present an overview of the existing techniques used in the synthesis of human postures and gestures. The third part is the detailed description of the proposed approach and methodology; the fourth section presents the results that have been obtained. We conclude with a discussion on the results and the future developments of the methodology.
2
State of the Art
In this section we present an overview of the different scientific developments focused on the study and generation of human postures and gestures. We distinguish two main trends: the biology (biomechanics studies) and the non-biology based (more computer-graphics-oriented) approach. 2.1 The biology based approach
Many biomechanical models have been developed to provide partial simulations of specific human movements: walking/running models, or simulations of dynamic postural control under unknown conditions [3], [4]. The biology based approach has been used in the research of biomechanical engineering, robotics and neurophysiology, to clarify the mechanisms of human walking [5], and modelling the muscular actuation systems of animals [6], [7] and human beings [8], [9]. The biologically inspired control architectures are one of the most recent trends in the field of robotics [10]. Models of the human nervous system, in conjunction with fuzzy logic, neural networks and evolutionary algorithms have been used to better understand and control the way humans and animals move and execute typical actions such as walking, running, reaching or grasping [11] [12], [4]. Some other studies focus on the analysis of the human motion to extract parametric models for posture recognition [13]. None of these developments has been specifically applied to create a general autonomous control system to drive the animation and behaviour of a virtual human in the framework of a virtual reality application. However, they can provide the basis for a general motion control model. 2.2 The computer graphics approach
In contrast with the above mentioned studies, the following citations are more related to the development of virtual reality and computer graphics applications in general. The most ”traditional” techniques used to synthesize human gestures include the use of kinematics, dynamics or a combination of them [14], citeko1996. Inverse Kinematics has been used for animation of complex articulated figures such as the human body, balance control, motion and postures correction are some of their applications [15], [16], [17]. These techniques are focused on the correction of predefined postures or common actions such as walking to make them more natural; however, they require specific goals to be predefined and don’t consider spontaneous movements. They can be used as tools for animators but don’t provide a system for autonomous motion. Another approach is the use of statistical analysis of observation data acquired by different means: motion capture systems, video or photographs. These methods use in different ways a database of pre-recorded movements to mix them, readapt them and reuse them in a variety of environments where the same kind of actions are required. Again, the main type of movements that are
studied are walking sequences and standard combinations of movements: walksit, sit-walk, etc. [1], [18], [19]. These approaches are not very suitable to be used as part of a system for the automatic generation of gestures. Synthesizing autonomous gestures is related to the area of behavioural animation. By behavioural animation we refer to the techniques applied to the synthesis of autonomous animation of virtual characters. Autonomy is one of the most priced goals in the field of character animation. Several studies and implementations have been done with the objective of giving a virtual character the ability to perceive its virtual environment and react to it by means of executing adequate tasks [20], [21], creating the illusion that the synthetic character is alive: it moves by itself and achieves relatively complex tasks. The subtle and spontaneous gestures are part of low-level behaviors, and they are essential to communicate any message and convey emotions. These ideas are just beginning to be explored by the scientific community, some studies focus on non-human characters, like dogs or other virtual animals [22]. One particularly difficult issue is the problem of what to do with the characters when there is no pre-defined task assigned to them. Research has been done to produce the so-called ”idle animation”: generating behavior to make the character move when there’s no specific action to do. Some work has focused on adding a pseudo-random ”perturbation” to an idle posture [23], avoiding ”frozen” characters. However, real people perform very different gestures when waiting for something or while attending or listening to someone. The gestures performed during ”idle states” depend on a variety of factors: emotional state, cultural background, the current situation (waiting for something, listening, or thinking), unexpected events (a change in the environment), etc. and can be classified as low-level behavior. Despite the advances in behavioural animation, autonomous virtual humans are not yet able to display the whole range of subtle gestures and mannerisms that characterize a real human being: facial expressions, body language, autonomous reactions, etc. Our work intends to advance the state of the art on the last category: providing virtual humans with the ability to react to unexpected events through reflex movements. After analyzing some of the existing techniques applied to the control and synthesis of human gestures, we observe a promising way in the biologically inspired systems, especially when the goal of autonomy and naturalness in the motions is the first priority. In the state of the art we observe that the metaphor of the nervous system has been used to simulate and control the motion of a specific limb with applications to medicine and/or robotics. We have followed this approach to build a general architecture for the synthesis of autonomous gestures. In the next section we describe our proposal in detail.
3
The virtual human neuromotor system
The main innovation of this work is the design of a distributed control architecture based on autonomous entities that intercommunicate with each other in a self-similar hierarchy (fractal architecture). We propose a control system inspired in the human nervous system which will be used to generate autonomous behaviour on virtual humans. Biologically based control systems have been applied mainly to simulations oriented to robotics or for biomechanics applications. The human nervous system (HNS) is divided into the central and peripheral nervous systems (CNS and PNS, respectively). The PNS consists of sensory neurons running from stimulus receptors that inform the CNS of the stimuli; and motor neurons running from the CNS to the muscles and glands - called effectors - that take action. The CNS consists of the spinal cord and the brain [24]. The CNS is a control centre which receives internal and external stimuli and sends orders to the effectors by means of the motor neurons. We are using this principle as the basic building block for a simplified model of the nervous system (NS) that will act as an animation controller for a virtual human. The proposed model of the Virtual Human NS is a distributed system capable to produce autonomous gestures (reflex movements) in reaction to a defined set of stimuli: external forces, temperature, and muscle effort. This NS model constitutes what we call the distributed animation control system (DACS). In conjunction with the DACS we have implemented a simplified model of the human locomotion system in order to provide the DACS with a set of effectors to move the different body parts. The Virtual Human Locomotion System (VHLS) models the human musculo-skeletal system as a skeleton structure, compliant with the H-Anim specification [25], whose joints can be moved by pairs of antagonist muscles, one pair for each degree of freedom. The implementation of the muscles gives importance to the effect they produce on the virtual humans joints, that’s why they are called effectors. The main idea behind the DACS is the definition of a minimum control entity constituted by three main sub entities or components: sensors, analyzers and effectors, defined as follows: The Sensors are the virtual devices capable of gathering information from the exterior world of the virtual human, such as temperature, contact with other objects, external forces, etc.; and also from the interior of the virtual human: stress, muscle effort, etc. They store the acquired information in the form of a stimulus vector, containing information on the intensity, direction and orientation of the received stimulus. The Analyzers are entities which concentrate information coming from one or many sensors or analyzers. An analyzer’s main function is to calculate the cumulative effect of the sensors attached to it. The reaction vector calculated by the analyzer is used by the effectors to generate the adequate reaction as a function of the intensity, direction and orientation of the stimulus received. The Effectors: these entities are black boxes capable to affect one or more degrees of freedom of the virtual human joints to which they’re attached. Currently, they’re implemented as inverse kinematics controllers that calculate the
position of the joints as a function of the reaction vector calculated by the analyzer attached to them. The components of the minimum control entity -sensors, analyzers and effectorswhich will constitute the basic building block of the DACS are shown on figure 1. This scheme will be replicated at different hierarchic levels providing main control centres (analyzers which will be linked to lower level analyzers) capable of taking high level decisions. The control entity is capable to receive (sense) a set of stimuli or information, process them and generate a reaction, which can be an order to an effector (a muscle or set of muscles) or a message to another control entity in the same or in a different hierarchic level. Each control entity (c.e.) is usually responsible of controlling one limb of the virtual human (arms, legs). The basic c.e. will be reproduced in different hierarchic levels in order to coordinate the whole body. The figure 1 illustrates this idea: a hierarchy of control entities following a fractal like structure (self-similar hierarchy). Each component works in an independent way, this characteristic allows for distributing the system.
Fig. 1. The basic control entity and the hierarchical structure of the DACS.
The communication between control entities will takes place depending on the intensity of the received stimuli, this will emulate the neurotransmitters effect (the chemical substances which act as communication channels between neurons) and allow or avoid communication at different levels. The simulation of the neurotransmitters effect will allow for modifying the intensity of the response to a given stimulus. For example, if the virtual human touches a hot object such as the grill of a stove, it will react with different speed or intensity depending on the actual temperature, it’s different to touch a grill at 20C than trying to do it when the object is at 150C. Spontaneous behaviour can be generated depending on the overall conditions of the virtual human. Spontaneous movements such as balancing the arms or changing the body weight from one leg to another in a
stand up posture, vary depending on the muscular and/or mental stress. People display different gestures when they stop after running or walking depending on the amount of energy they have spent, among many other factors. These gestures can be also considered as reactions to a certain kind of stimuli as well. Until now we have explained the general principles of the DACS which will constitute the nervous and motor system and provide autonomous animation to a virtual human, depending on the internal and external stimuli and information coming from its environment. In the next section we describe a test application that shows the feasibility of the proposed model. 3.1 Test application: reaction to thermic stimuli
To show the feasibility of using the Distributed Animation Control System as the neuromotor system for a virtual human, we have implemented a demonstration application which generates reflex movements for the arm as reaction to thermic stimuli. The virtual human will stand in front of a stove with its left hand over one of the burners. The temperature of the burner will be modified in the different tests. The objective is to see different levels of reaction depending on the perceived temperature. The reaction levels have been classified intro 3 main regions depending on the stimulus intensity: green zone -no reaction is required-, yellow zone -controlled reaction, the speed of the motion starts to increase depending on the stimulus intensity-, red zone -the reaction is more ”violent” since the stimulus intensity reaches the highest values, in this level the joint limits are usually reached and a ”bounding” motion is produced as a reaction. If the temperature of the burner is above 40C, the reflex movements start to appear, ranging from a slight movement of the wrist to separate the hand from the burner level, up to a violent fast movement involving all the arm joints in order to retire the hand as soon as possible if the temperature raises to values around the 100C. We used an H-Anim compliant virtual human model and implemented an animation control for one arm (see figure 2), with temperature sensors (to receive external stimuli) in the palm of the hand. The general algorithm used to calculate the reflex movement is the following: Analyzers and sensors are arranged in a tree structure. The sensors are sampled at each animation frame. The analyzer-sensor tree is traversed in post-order (first children, then the local parent). Allowing each analyzer to gather the information of the sensors attached to it. The process is recursively repeated letting the analyzers concentrate the information coming from the lower levels. Each segment of the articulated character has a main analyzer. Segment analyzers are associated to the main limb analyzer, the one for the arm in this case. The limb analyzers are usually the local root of the sensor-analyzer tree and contain the overall reaction vector specifying the velocity of the reflex movement. The reaction vector contains the orientation and direction to be followed by the limb in order to react as required -rejecting the thermic stimuli.
Fig. 2. The animation control for the arm.
The orientation and direction of the stimulus vector depend on the position of the sensor relative to the stimulus source -orientation is inverted representing the need for rejecting the stimulus source. The stimulus vectors are calculated as follows: the vector magnitude -stimulus intensity- is a function of the Euclidean distance between the sensor and the stimulus source -represented by a point in the 3D space. The intensity decreases with the distance -in our tests this is done in a linear way. The analyzers compute the reaction vector by calculating a vector addition of the stimulus vectors from their associated sensors. The actual movement of the arm is calculated using inverse kinematics (IK) [26], [27]. The end effector required by the IK algorithm is the segment receiving the highest stimulus intensity -magnitude of the reaction vector-, as calculated by the segment analyzers. The trajectory to be followed by the limb -end effector- is defined by the reaction vector computed by the main limb analyzer. The motion speed is directly proportional to the intensity of the reaction vector. As explained in the previous section, the reaction speed can range from no reaction at all when intensity falls into the tolerance interval- up to a very fast reaction causing the limb to reach the joint limits very fast, even before the sensors detect there is no need to reject the stimulus source anymore. In the later case, the effector nodes -not shown in figure 2, but associated to each joint, force the limb to recover a comfortable posture -under the joint limits. 3.2 Technical details of the implementation
The demonstration application has been implemented as a java applet. The 3D animation is done using a 3D rendering engine for java [28]. The components of the basic control entity are java classes which extend the thread class in order to be instantiated and run as independent threads. The analyzer objects monitor continuously their attached components (effectors, sensors or other analyzers)
and establish the required communications. The virtual human model is an HAnim VRML file, the stove is a conventional VRML’97 file. The demonstration was run on a PC workstation with a bi-Xeon at 1.2Mhz processor and 1Gb of RAM using MS-Windows 2000. The animation runs at 30 frames per second and the gives the impression of a natural reaction speed. The figure 3 shows different levels of reaction depending on the preset temperature. For each test the simulation is reset to the initial posture of the virtual human with its left hand over the stove burner.
Fig. 3. Snapshots of the test application.
4
Conclusions
The work presented in this paper is still in an early stage and many more stimuliresponse pairs must be modelled. One of the most important questions to solve is the modelling of internal stimuli such as emotions, stress and other cognitive processes. Our research is not advanced enough to give a precise answer at this moment. Nevertheless, the arm implementation with sensors and effectors driven by the fractal hierarchy of basic control entities has shown the feasibility of this system architecture to be implemented at larger scale (full body control). The test application provides a way for the virtual human to react in an automatic way to a certain kind of stimuli in the virtual environment. One of the main
drawbacks of our current model is the fact that the reflex movements are deterministic; to a given stimulus there is always the same reaction. A higher level control must be put in place to take into account the stress induced by previous stimuli and the variations it produces on subsequent gestures. In future developments, a central control unit will be implemented to emulate the brain as a main control centre and reach a higher level of autonomy: based not only on reflex gestures, but able to generate more complex behaviour driven by high level directives or intentions. Implementing virtual senses such as vision or audition will let us synthesize more advanced reactions, e.g. the virtual human could be able to raise its arms to protect itself from an object being thrown toward it. This kind of reflex movement would require a set of rules for selecting the kind of gesture in function of the stimulus being received -e.g. detection of an object being thrown towards us triggers a predefined defensive gesture of the arms to protect the body part being menaced. We believe this kind of behaviour can be generated using the virtual neuromotor system we are proposing by means of modelling the gestures as reactions to internal/external stimuli.
5
Acknowledgements
Maya licence used for modeling the character has been granted by Alias through their research donation program.
References
1. Badler, N., Chi, D., Chopra-Khullar, S.: Virtual human animation based on movement observation and cognitive behavior models. In: Computer Animation 1999. (1999) 128–137 2. Perlin, K.: Building virtual actors who can really act. In: 2nd International Conference on Virtual Storytelling. (2003) 3. Gorce, P.: Dynamic postural control method for biped in unknown environment. In: IEEE Transactions on Systems, Man and Cybernetics. (1999) 616–626 4. Kubica, E., Wang, D., Winter, D.: Feedforward and deterministic fuzzy control of balance and posture during human gait. In: IEEE International Conference on Robotics and Automation. (2001) 2293–2298 5. Ok, S., Miyashita, K., Hase, K.: Evolving bipedal locomotion with genetic programming - a preliminary report. In: Congress on Evolutionary Computation. (2001) 1025–1032 6. He, J., Levine, W., Loeb, G.: The modeling of the neuro-musculo-skeletal control system of a cat hindlimb. In: IEEE International Symposium on Intelligent Control. (1988) 406–411 7. Garcia-Cordova, F., Guerrero-Gonzalez, A., Pedreno-Molina, J., Moran, J.: Emulation of the animal muscular actuation system in an experimental platform. In: IEEE International Conference on Systems, Man, and Cybernetics. (2001) 64–69 8. Brown, I., Loeb, G.: Design of a mathematical model of force in whole skeletal muscle. In: IEEE 17th Annual Conference on Engineering in Medicine and Biology Society. (1995) 1243–1244
9. Guihard, M., Gorce, P.: Dynamic control of an artificial muscle arm. In: IEEE Transactions on Systems, Man and Cybernetics. (1999) 813–818 10. Northrup, S., Sarkar, N., Kawamura, K.: Biologically-inspired control architecture for a humanoid robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. (2001) 1100–1105 11. Karniel, A., Inbar, G.: A model for learning human reaching-movements. In: 18th Annual International Conference of the IEEE on Engineering in Medicine and Biology Society. (1996) 619–620 12. Andry, P., Gaussier, P., Moga, S., Banquet, J., Nadel, J.: Learning and communication in imitation: An autonomous robot perspective. In: IEEE Transaction on Systems, Man and Cybernetics, Part A: Systems and Humans. (2001) 431–444 13. Hu, C., Yu, Q., Li, Y., Ma, S.: Extraction of parametric human model for posture recognition using genetic algorithm. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition. (2000) 518–523 14. Kacic-Alesic, Z., Nordenstam, M., Bullock, D.: A practical dynamics system. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Eurographics Association (2003) 7–16 15. Boulic, R., Mas, R.: Hierarchical kinematics behaviors for complex articulated figures. In: Interactive Computer Animation, Prentice Hall. (1996) 40–70 16. Rodriguez, I., Peinado, M., Boulic, R., Meziat, D.: Reaching volumes generated by means of octal trees and cartesian constraints. In: CGI 2003. (2003) 17. Komura, T., Kuroda, A., Kudoh, S., Lan, T., Shinagawa, Y.: An inverse kinematics method for 3d figures with motion data. In: Computer Graphics International, 2003. (2003) 242–247 18. Tanco, L., Hilton, A.: Realistic synthesis of novel human movements from a database of motion capture examples. In: Workshop on Human Motion. (2000) 137–142 19. Ashida, K., Lee, S.J., Allbeck, J., Sun, H., Badler, N., Metaxas, D.: Pedestrians: Creating agent behaviors through statistical analysis of observation data. In: Computer Animation 2001. (2001) 20. Blumberg, B., Galyean, T.: Multi-level direction of autonomous creatures for realtime virtual environments. In: SIGGRAPH 1995. (1995) 21. Tu, X., Terzopoulos, D.: Artificial fishes: physics, locomotion, perception, behavior. In: Proceedings of the 21st annual conference on Computer graphics and interactive techniques, ACM Press (1994) 43–50 22. Isla, D., Blumberg, B.: Object persistence for synthetic creatures. In: International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS). (2002) 23. Perlin, K.: Real time responsive animation with personality. In: IEEE Transactions on Visualization and Computer Graphics. (1995) 5–15 24. Kimball, J.: (Organization of the Nervous System, Kimball’s Biology Pages. 2003) http://users.rcn.com/jkimball.ma.ultranet/BiologyPages. 25. H-Anim: (The humanoid animation working group. http://www.h-anim.org) 26. Deepak, T., Goswami, A., Badler, N.: Real-time inverse kinematics techniques for anthropomorphic limbs. In: Graphical Models and Image Processing. (2000) 27. IKAN: (Inverse kinematics using analytical methods. http://hms.upenn.edu/software/ik/software.html) 28. Shout3D: (Shout3D, Eyematic Interfaces Inc. (2001)) http://www.shout3d.com.
Mario Gutierrez, Frederic Vexo and Daniel Thalmann
Virtual Reality Laboratory (VRlab) Swiss Federal Institute of Technology Lausanne (EPFL) Lausanne Switzerland CH-1015 {mario.gutierrez,frederic.vexo,daniel.thalmann}@epfl.ch
Abstract. This paper presents the results of a method to produce autonomous animation of virtual humans. In particular, the proposed methodology is focused on the autonomous synthesis of non-voluntary gestures such as reflexes and subtle movements which provide a noticeable impression of realism and naturalness. The final goal of this technique is to produce virtual humans with a more spontaneous, non pre-programmed behaviour. For the moment, the technique is applied to the synthesis of reflex movements of the arm, in reaction to thermic stimuli. Nevertheless, a general architecture is outlined.
1
Introduction
The animation of virtual humans and computer animation in general have always searched to produce realistic imagery and true naturalness in the motions. One particularly difficult problem is to provide automatic animation of a virtual human displaying a behaviour that truly resembles a real human [1], [2]. Our goal is to animate virtual humans (body gestures) in an autonomous way, giving the illusion that the artificial human being is a living creature displaying spontaneous behaviour according to its internal information and external stimuli coming from its virtual environment. The proposed method is a real-time, distributed control system inspired in the human nervous system, which will be able to produce natural movements in response to external and internal stimuli. The advantage of the real-time control techniques over the ”Computer Graphics movie” approach is their promise of combined autonomy and realism as well as their focus on reusability. The autonomous control system we are presenting could be instantiated and used to animate several different characters in a variety of situations. This paper is organized according to the following structure: in the next section we present an overview of the existing techniques used in the synthesis of human postures and gestures. The third part is the detailed description of the proposed approach and methodology; the fourth section presents the results that have been obtained. We conclude with a discussion on the results and the future developments of the methodology.
2
State of the Art
In this section we present an overview of the different scientific developments focused on the study and generation of human postures and gestures. We distinguish two main trends: the biology (biomechanics studies) and the non-biology based (more computer-graphics-oriented) approach. 2.1 The biology based approach
Many biomechanical models have been developed to provide partial simulations of specific human movements: walking/running models, or simulations of dynamic postural control under unknown conditions [3], [4]. The biology based approach has been used in the research of biomechanical engineering, robotics and neurophysiology, to clarify the mechanisms of human walking [5], and modelling the muscular actuation systems of animals [6], [7] and human beings [8], [9]. The biologically inspired control architectures are one of the most recent trends in the field of robotics [10]. Models of the human nervous system, in conjunction with fuzzy logic, neural networks and evolutionary algorithms have been used to better understand and control the way humans and animals move and execute typical actions such as walking, running, reaching or grasping [11] [12], [4]. Some other studies focus on the analysis of the human motion to extract parametric models for posture recognition [13]. None of these developments has been specifically applied to create a general autonomous control system to drive the animation and behaviour of a virtual human in the framework of a virtual reality application. However, they can provide the basis for a general motion control model. 2.2 The computer graphics approach
In contrast with the above mentioned studies, the following citations are more related to the development of virtual reality and computer graphics applications in general. The most ”traditional” techniques used to synthesize human gestures include the use of kinematics, dynamics or a combination of them [14], citeko1996. Inverse Kinematics has been used for animation of complex articulated figures such as the human body, balance control, motion and postures correction are some of their applications [15], [16], [17]. These techniques are focused on the correction of predefined postures or common actions such as walking to make them more natural; however, they require specific goals to be predefined and don’t consider spontaneous movements. They can be used as tools for animators but don’t provide a system for autonomous motion. Another approach is the use of statistical analysis of observation data acquired by different means: motion capture systems, video or photographs. These methods use in different ways a database of pre-recorded movements to mix them, readapt them and reuse them in a variety of environments where the same kind of actions are required. Again, the main type of movements that are
studied are walking sequences and standard combinations of movements: walksit, sit-walk, etc. [1], [18], [19]. These approaches are not very suitable to be used as part of a system for the automatic generation of gestures. Synthesizing autonomous gestures is related to the area of behavioural animation. By behavioural animation we refer to the techniques applied to the synthesis of autonomous animation of virtual characters. Autonomy is one of the most priced goals in the field of character animation. Several studies and implementations have been done with the objective of giving a virtual character the ability to perceive its virtual environment and react to it by means of executing adequate tasks [20], [21], creating the illusion that the synthetic character is alive: it moves by itself and achieves relatively complex tasks. The subtle and spontaneous gestures are part of low-level behaviors, and they are essential to communicate any message and convey emotions. These ideas are just beginning to be explored by the scientific community, some studies focus on non-human characters, like dogs or other virtual animals [22]. One particularly difficult issue is the problem of what to do with the characters when there is no pre-defined task assigned to them. Research has been done to produce the so-called ”idle animation”: generating behavior to make the character move when there’s no specific action to do. Some work has focused on adding a pseudo-random ”perturbation” to an idle posture [23], avoiding ”frozen” characters. However, real people perform very different gestures when waiting for something or while attending or listening to someone. The gestures performed during ”idle states” depend on a variety of factors: emotional state, cultural background, the current situation (waiting for something, listening, or thinking), unexpected events (a change in the environment), etc. and can be classified as low-level behavior. Despite the advances in behavioural animation, autonomous virtual humans are not yet able to display the whole range of subtle gestures and mannerisms that characterize a real human being: facial expressions, body language, autonomous reactions, etc. Our work intends to advance the state of the art on the last category: providing virtual humans with the ability to react to unexpected events through reflex movements. After analyzing some of the existing techniques applied to the control and synthesis of human gestures, we observe a promising way in the biologically inspired systems, especially when the goal of autonomy and naturalness in the motions is the first priority. In the state of the art we observe that the metaphor of the nervous system has been used to simulate and control the motion of a specific limb with applications to medicine and/or robotics. We have followed this approach to build a general architecture for the synthesis of autonomous gestures. In the next section we describe our proposal in detail.
3
The virtual human neuromotor system
The main innovation of this work is the design of a distributed control architecture based on autonomous entities that intercommunicate with each other in a self-similar hierarchy (fractal architecture). We propose a control system inspired in the human nervous system which will be used to generate autonomous behaviour on virtual humans. Biologically based control systems have been applied mainly to simulations oriented to robotics or for biomechanics applications. The human nervous system (HNS) is divided into the central and peripheral nervous systems (CNS and PNS, respectively). The PNS consists of sensory neurons running from stimulus receptors that inform the CNS of the stimuli; and motor neurons running from the CNS to the muscles and glands - called effectors - that take action. The CNS consists of the spinal cord and the brain [24]. The CNS is a control centre which receives internal and external stimuli and sends orders to the effectors by means of the motor neurons. We are using this principle as the basic building block for a simplified model of the nervous system (NS) that will act as an animation controller for a virtual human. The proposed model of the Virtual Human NS is a distributed system capable to produce autonomous gestures (reflex movements) in reaction to a defined set of stimuli: external forces, temperature, and muscle effort. This NS model constitutes what we call the distributed animation control system (DACS). In conjunction with the DACS we have implemented a simplified model of the human locomotion system in order to provide the DACS with a set of effectors to move the different body parts. The Virtual Human Locomotion System (VHLS) models the human musculo-skeletal system as a skeleton structure, compliant with the H-Anim specification [25], whose joints can be moved by pairs of antagonist muscles, one pair for each degree of freedom. The implementation of the muscles gives importance to the effect they produce on the virtual humans joints, that’s why they are called effectors. The main idea behind the DACS is the definition of a minimum control entity constituted by three main sub entities or components: sensors, analyzers and effectors, defined as follows: The Sensors are the virtual devices capable of gathering information from the exterior world of the virtual human, such as temperature, contact with other objects, external forces, etc.; and also from the interior of the virtual human: stress, muscle effort, etc. They store the acquired information in the form of a stimulus vector, containing information on the intensity, direction and orientation of the received stimulus. The Analyzers are entities which concentrate information coming from one or many sensors or analyzers. An analyzer’s main function is to calculate the cumulative effect of the sensors attached to it. The reaction vector calculated by the analyzer is used by the effectors to generate the adequate reaction as a function of the intensity, direction and orientation of the stimulus received. The Effectors: these entities are black boxes capable to affect one or more degrees of freedom of the virtual human joints to which they’re attached. Currently, they’re implemented as inverse kinematics controllers that calculate the
position of the joints as a function of the reaction vector calculated by the analyzer attached to them. The components of the minimum control entity -sensors, analyzers and effectorswhich will constitute the basic building block of the DACS are shown on figure 1. This scheme will be replicated at different hierarchic levels providing main control centres (analyzers which will be linked to lower level analyzers) capable of taking high level decisions. The control entity is capable to receive (sense) a set of stimuli or information, process them and generate a reaction, which can be an order to an effector (a muscle or set of muscles) or a message to another control entity in the same or in a different hierarchic level. Each control entity (c.e.) is usually responsible of controlling one limb of the virtual human (arms, legs). The basic c.e. will be reproduced in different hierarchic levels in order to coordinate the whole body. The figure 1 illustrates this idea: a hierarchy of control entities following a fractal like structure (self-similar hierarchy). Each component works in an independent way, this characteristic allows for distributing the system.
Fig. 1. The basic control entity and the hierarchical structure of the DACS.
The communication between control entities will takes place depending on the intensity of the received stimuli, this will emulate the neurotransmitters effect (the chemical substances which act as communication channels between neurons) and allow or avoid communication at different levels. The simulation of the neurotransmitters effect will allow for modifying the intensity of the response to a given stimulus. For example, if the virtual human touches a hot object such as the grill of a stove, it will react with different speed or intensity depending on the actual temperature, it’s different to touch a grill at 20C than trying to do it when the object is at 150C. Spontaneous behaviour can be generated depending on the overall conditions of the virtual human. Spontaneous movements such as balancing the arms or changing the body weight from one leg to another in a
stand up posture, vary depending on the muscular and/or mental stress. People display different gestures when they stop after running or walking depending on the amount of energy they have spent, among many other factors. These gestures can be also considered as reactions to a certain kind of stimuli as well. Until now we have explained the general principles of the DACS which will constitute the nervous and motor system and provide autonomous animation to a virtual human, depending on the internal and external stimuli and information coming from its environment. In the next section we describe a test application that shows the feasibility of the proposed model. 3.1 Test application: reaction to thermic stimuli
To show the feasibility of using the Distributed Animation Control System as the neuromotor system for a virtual human, we have implemented a demonstration application which generates reflex movements for the arm as reaction to thermic stimuli. The virtual human will stand in front of a stove with its left hand over one of the burners. The temperature of the burner will be modified in the different tests. The objective is to see different levels of reaction depending on the perceived temperature. The reaction levels have been classified intro 3 main regions depending on the stimulus intensity: green zone -no reaction is required-, yellow zone -controlled reaction, the speed of the motion starts to increase depending on the stimulus intensity-, red zone -the reaction is more ”violent” since the stimulus intensity reaches the highest values, in this level the joint limits are usually reached and a ”bounding” motion is produced as a reaction. If the temperature of the burner is above 40C, the reflex movements start to appear, ranging from a slight movement of the wrist to separate the hand from the burner level, up to a violent fast movement involving all the arm joints in order to retire the hand as soon as possible if the temperature raises to values around the 100C. We used an H-Anim compliant virtual human model and implemented an animation control for one arm (see figure 2), with temperature sensors (to receive external stimuli) in the palm of the hand. The general algorithm used to calculate the reflex movement is the following: Analyzers and sensors are arranged in a tree structure. The sensors are sampled at each animation frame. The analyzer-sensor tree is traversed in post-order (first children, then the local parent). Allowing each analyzer to gather the information of the sensors attached to it. The process is recursively repeated letting the analyzers concentrate the information coming from the lower levels. Each segment of the articulated character has a main analyzer. Segment analyzers are associated to the main limb analyzer, the one for the arm in this case. The limb analyzers are usually the local root of the sensor-analyzer tree and contain the overall reaction vector specifying the velocity of the reflex movement. The reaction vector contains the orientation and direction to be followed by the limb in order to react as required -rejecting the thermic stimuli.
Fig. 2. The animation control for the arm.
The orientation and direction of the stimulus vector depend on the position of the sensor relative to the stimulus source -orientation is inverted representing the need for rejecting the stimulus source. The stimulus vectors are calculated as follows: the vector magnitude -stimulus intensity- is a function of the Euclidean distance between the sensor and the stimulus source -represented by a point in the 3D space. The intensity decreases with the distance -in our tests this is done in a linear way. The analyzers compute the reaction vector by calculating a vector addition of the stimulus vectors from their associated sensors. The actual movement of the arm is calculated using inverse kinematics (IK) [26], [27]. The end effector required by the IK algorithm is the segment receiving the highest stimulus intensity -magnitude of the reaction vector-, as calculated by the segment analyzers. The trajectory to be followed by the limb -end effector- is defined by the reaction vector computed by the main limb analyzer. The motion speed is directly proportional to the intensity of the reaction vector. As explained in the previous section, the reaction speed can range from no reaction at all when intensity falls into the tolerance interval- up to a very fast reaction causing the limb to reach the joint limits very fast, even before the sensors detect there is no need to reject the stimulus source anymore. In the later case, the effector nodes -not shown in figure 2, but associated to each joint, force the limb to recover a comfortable posture -under the joint limits. 3.2 Technical details of the implementation
The demonstration application has been implemented as a java applet. The 3D animation is done using a 3D rendering engine for java [28]. The components of the basic control entity are java classes which extend the thread class in order to be instantiated and run as independent threads. The analyzer objects monitor continuously their attached components (effectors, sensors or other analyzers)
and establish the required communications. The virtual human model is an HAnim VRML file, the stove is a conventional VRML’97 file. The demonstration was run on a PC workstation with a bi-Xeon at 1.2Mhz processor and 1Gb of RAM using MS-Windows 2000. The animation runs at 30 frames per second and the gives the impression of a natural reaction speed. The figure 3 shows different levels of reaction depending on the preset temperature. For each test the simulation is reset to the initial posture of the virtual human with its left hand over the stove burner.
Fig. 3. Snapshots of the test application.
4
Conclusions
The work presented in this paper is still in an early stage and many more stimuliresponse pairs must be modelled. One of the most important questions to solve is the modelling of internal stimuli such as emotions, stress and other cognitive processes. Our research is not advanced enough to give a precise answer at this moment. Nevertheless, the arm implementation with sensors and effectors driven by the fractal hierarchy of basic control entities has shown the feasibility of this system architecture to be implemented at larger scale (full body control). The test application provides a way for the virtual human to react in an automatic way to a certain kind of stimuli in the virtual environment. One of the main
drawbacks of our current model is the fact that the reflex movements are deterministic; to a given stimulus there is always the same reaction. A higher level control must be put in place to take into account the stress induced by previous stimuli and the variations it produces on subsequent gestures. In future developments, a central control unit will be implemented to emulate the brain as a main control centre and reach a higher level of autonomy: based not only on reflex gestures, but able to generate more complex behaviour driven by high level directives or intentions. Implementing virtual senses such as vision or audition will let us synthesize more advanced reactions, e.g. the virtual human could be able to raise its arms to protect itself from an object being thrown toward it. This kind of reflex movement would require a set of rules for selecting the kind of gesture in function of the stimulus being received -e.g. detection of an object being thrown towards us triggers a predefined defensive gesture of the arms to protect the body part being menaced. We believe this kind of behaviour can be generated using the virtual neuromotor system we are proposing by means of modelling the gestures as reactions to internal/external stimuli.
5
Acknowledgements
Maya licence used for modeling the character has been granted by Alias through their research donation program.
References
1. Badler, N., Chi, D., Chopra-Khullar, S.: Virtual human animation based on movement observation and cognitive behavior models. In: Computer Animation 1999. (1999) 128–137 2. Perlin, K.: Building virtual actors who can really act. In: 2nd International Conference on Virtual Storytelling. (2003) 3. Gorce, P.: Dynamic postural control method for biped in unknown environment. In: IEEE Transactions on Systems, Man and Cybernetics. (1999) 616–626 4. Kubica, E., Wang, D., Winter, D.: Feedforward and deterministic fuzzy control of balance and posture during human gait. In: IEEE International Conference on Robotics and Automation. (2001) 2293–2298 5. Ok, S., Miyashita, K., Hase, K.: Evolving bipedal locomotion with genetic programming - a preliminary report. In: Congress on Evolutionary Computation. (2001) 1025–1032 6. He, J., Levine, W., Loeb, G.: The modeling of the neuro-musculo-skeletal control system of a cat hindlimb. In: IEEE International Symposium on Intelligent Control. (1988) 406–411 7. Garcia-Cordova, F., Guerrero-Gonzalez, A., Pedreno-Molina, J., Moran, J.: Emulation of the animal muscular actuation system in an experimental platform. In: IEEE International Conference on Systems, Man, and Cybernetics. (2001) 64–69 8. Brown, I., Loeb, G.: Design of a mathematical model of force in whole skeletal muscle. In: IEEE 17th Annual Conference on Engineering in Medicine and Biology Society. (1995) 1243–1244
9. Guihard, M., Gorce, P.: Dynamic control of an artificial muscle arm. In: IEEE Transactions on Systems, Man and Cybernetics. (1999) 813–818 10. Northrup, S., Sarkar, N., Kawamura, K.: Biologically-inspired control architecture for a humanoid robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. (2001) 1100–1105 11. Karniel, A., Inbar, G.: A model for learning human reaching-movements. In: 18th Annual International Conference of the IEEE on Engineering in Medicine and Biology Society. (1996) 619–620 12. Andry, P., Gaussier, P., Moga, S., Banquet, J., Nadel, J.: Learning and communication in imitation: An autonomous robot perspective. In: IEEE Transaction on Systems, Man and Cybernetics, Part A: Systems and Humans. (2001) 431–444 13. Hu, C., Yu, Q., Li, Y., Ma, S.: Extraction of parametric human model for posture recognition using genetic algorithm. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition. (2000) 518–523 14. Kacic-Alesic, Z., Nordenstam, M., Bullock, D.: A practical dynamics system. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Eurographics Association (2003) 7–16 15. Boulic, R., Mas, R.: Hierarchical kinematics behaviors for complex articulated figures. In: Interactive Computer Animation, Prentice Hall. (1996) 40–70 16. Rodriguez, I., Peinado, M., Boulic, R., Meziat, D.: Reaching volumes generated by means of octal trees and cartesian constraints. In: CGI 2003. (2003) 17. Komura, T., Kuroda, A., Kudoh, S., Lan, T., Shinagawa, Y.: An inverse kinematics method for 3d figures with motion data. In: Computer Graphics International, 2003. (2003) 242–247 18. Tanco, L., Hilton, A.: Realistic synthesis of novel human movements from a database of motion capture examples. In: Workshop on Human Motion. (2000) 137–142 19. Ashida, K., Lee, S.J., Allbeck, J., Sun, H., Badler, N., Metaxas, D.: Pedestrians: Creating agent behaviors through statistical analysis of observation data. In: Computer Animation 2001. (2001) 20. Blumberg, B., Galyean, T.: Multi-level direction of autonomous creatures for realtime virtual environments. In: SIGGRAPH 1995. (1995) 21. Tu, X., Terzopoulos, D.: Artificial fishes: physics, locomotion, perception, behavior. In: Proceedings of the 21st annual conference on Computer graphics and interactive techniques, ACM Press (1994) 43–50 22. Isla, D., Blumberg, B.: Object persistence for synthetic creatures. In: International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS). (2002) 23. Perlin, K.: Real time responsive animation with personality. In: IEEE Transactions on Visualization and Computer Graphics. (1995) 5–15 24. Kimball, J.: (Organization of the Nervous System, Kimball’s Biology Pages. 2003) http://users.rcn.com/jkimball.ma.ultranet/BiologyPages. 25. H-Anim: (The humanoid animation working group. http://www.h-anim.org) 26. Deepak, T., Goswami, A., Badler, N.: Real-time inverse kinematics techniques for anthropomorphic limbs. In: Graphical Models and Image Processing. (2000) 27. IKAN: (Inverse kinematics using analytical methods. http://hms.upenn.edu/software/ik/software.html) 28. Shout3D: (Shout3D, Eyematic Interfaces Inc. (2001)) http://www.shout3d.com.
