Patent Application
20170131775
An embodiment of the present invention is generally related to haptic feedback in the context of virtual reality systems and augmented reality systems.
Many virtual and augmented reality systems exist in which contact with a virtual object, as seen by a user, is accompanied with a sense of touch (haptic output) so as to give a better impression of immersion. Such systems usually include “gloves” or “suits” or other contact means that provide a stimulation or force feedback at the user body location that is virtually contacting said virtual object. Providing said stimulation point is difficult in situations where the user cannot, or does not wish to, wear any such apparatus.
As illustrated
However, a haptic feedback glove has many potential drawbacks. For example, feedback gloves can be hot and uncomfortable during extended use. Moreover, some users may find using the gloves uncomfortable during extend use due to health reasons, such as users with arthritis. High performance haptic feedback gloves can be more expensive than desired in some applications.
Embodiments of the present invention were developed in view of some of these and other issues.
The foregoing summary, as well as the following detailed description of illustrative implementations, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the implementations, there is shown in the drawings example constructions of the implementations; however, the implementations are not limited to the specific methods and instrumentalities disclosed. In the drawings:
Embodiments of the present invention provide haptic feedback by referral of sensation from one body location to another. That is, stimulation/sensation is provided in a first body location and the user's brain, through subliminal or formal training, learns to associate that with a second location. The training process utilizes the plasticity of human proprioception, which is a form of neural plasticity.
The referred haptic feedback can be coordinated with virtual images generated by a head mounted display (HMD), such as virtual reality (VR) or augmented reality (AR) HMD. In one embodiment, haptic feedback is provided by generating stimulation/sensation in a region, other than the fingers of a human hand, and through training users learn to associate the haptic feedback with their fingers. Other examples include using referral of sensation to provide haptic feedback for feet, legs, arms, etc.
Embodiments of the present invention include systems having a head mounted display and a separate haptic device. Alternatively, the haptic device may be integrated into the head mounted display. For example, in one embodiment the plasticity of human proprioception and other somatosensory pathways permits the redirection the physical sensation of touch in the area of a head mounted display (for example), to a body part as “seen” interacting with a virtual object. As another example, a head mounted virtual reality system may induce a touch sensation at a user's face or other head location that is visually correlated to the pushing of a button, or contact with a specific surface generated in the virtual space. After repeated experience of these correlated events, users may begin to experience the touch sensation as originating in the body part “contacting” the object or surface in a virtual space.
The haptic actuators 203 may comprise different numbers and arrangements of actuators depending on how many different stimulations/sensations are required to form an association with a specific body part. The haptic actuators 203 may be a set of spaced apart actuators such that associations may be made in the user's brain when specific actuators are active. Moreover, an individual actuator may have a range of output settings (e.g., frequency, duty cycle, amplitude) such that specific actuation patterns may be generated in an individual actuator or in set of actuators.
The haptic actuators 203 may be implemented in different ways. Touch actuators may make use of vibration, electric stimulation, acoustic stimulation, hot or cold spots, puffs of air pressure, mechanical means pressing on the skin or hair, etc. For example, in a game in which there is a penalty for touching some “electrified” objects, a small electric shock may be delivered when the virtual object is touched. In another embodiment, the HMD may tighten its band around the user's head as the user applies pressure with a virtual tool upon a virtual work piece. Electro-mechanical means such as linear motors or shape memory or electroactive polymer actuators may be used to move a contacting means or stylus along a path on the user, corresponding to some virtual movement or amount of force, etc. in the virtual world. Miniature vibrating motors with unbalanced rotors may be used to provide touch sensation or shake the visual display according to software generated events. In some embodiments the proximity of the HMD to specific areas of the user's brain may allow the use of transcranial magnetic stimulation or other electromagnetic or acoustic energy to generate direct brain stimulation. It will also be understood that combinations or sub-combination of different actuator types may be employed. Additionally, the haptic actuators may be employed as units that include a control processor, memory, communication interface, and any other required electronics to support the operation of the haptic actuators.
In one embodiment, the HMD communicates the actuation control signals to the haptic actuation unit 303 by direct wire (not shown) or RF link or near field or skin conductance means, etc. In one embodiment, the sensations produced by the remote haptic actuation unit 303 are generated in response to interactions between virtual objects or surfaces and body locations that are not necessarily at the same position on the body as that remote unit.
Whereas the remote haptic actuation unit 303 of
In one embodiment, the haptic actuators provide stimulation in or around the human ear. The human ear provides a sensitive location for haptic stimulation that can be then referred to seem to be originating in other parts of the body. In one embodiment of the current invention, points on the outer ear or inside the ear canal may be stimulated by haptic actuators and, through training, be neurologically mapped to other parts of the body.
Examples of training will now be described. The generations of a referred haptic response depends on forming an association in the user's brain between the location of the physical stimulus and the referred location. Training is likely necessary to establish this association. In principal it might be possible, in some cases, to provide subliminal training. For example, suppose that a haptic actuation unit 303 is located on a user's arm. During the play of a game, the unit may be activated when a user's hand interacts with a virtual object. In this way, during the play of a game a user is exposed to haptic stimulation in one part of their body and experiences interactions with virtual objects that subliminally form associations to create the referred haptic response. Additionally, the level of haptic stimulation from the haptic actuators could be gradually increased during game play to gradually train the user.
Although it is possible for the training to happen subliminally during game play, it is also possible to provide direct training activities. An example of a simple initial training method is shown in
The training proceeds by programmatically choosing a virtual object at random and moving 535 that object so that the user “sees” it touch an associated fingertip while the system activates the referred haptic stimulation 540 that is to be mentally mapped to that fingertip. The stimulation is stopped 545 and the virtual object is returned to a proximal but stationary position. This operation may be performed at least once. However, more generally it may be repeated many times for a session. Each hand, right and left, may be trained in subsequent sessions. Additionally, the training may be performed with different positions, such as both palm up and palm down.
It will be understood that the training may be perform in a sequence. For example, in one embodiment the training of
Training may be organized as a game, itself.
It will be understood that
The training shown in
Training methods for this kind of interaction can be designed along the lines shown above, but with an added means of measuring and providing feedback on the degree of interaction.
In one embodiment, variable finger force training is provided. An example is shown in
While
The training system may, in theory, be provided with the HMD or be in a computing device (not shown in
As previously discussed, embodiments of the present invention may be applied to both AR and VR environments. Additionally, embodiments of the present invention may be applied to AR environments in which there are additional real game elements.
While examples have been described in which a user touches a virtual object with their hand, it will also be understand that in various types of AR and VR games that users may “hold” or “grasp” virtual objects, such as virtual swords, magic wands, etc. The referred haptic feedback may thus also be used to refer to sensations associated with AR and VR fantasy elements grasped by a user's hand.
Whereas embodiments of the invention have been described in terms of matching touch to vision, those skilled in the art will understand that the referred touch may be matched to auditory, proprioception, or other senses. Those familiar with virtual reality immersion games will understand that avatars created for users may have anatomical parts that do not correspond to human form (tails, extra arms etc.) that, nonetheless, may be trained and mapped to referred haptic simulation. This expansion of the internal “body image” in the mind greatly enhances the immersive experience of this category of game play.
It will also be understood that in some embodiments aspects of the haptic stimulation may be customized (“tuned”) for an individual user. For example, individual human beings have a different degree of skin sensitivity due to the structure of the skin and surrounding tissues affecting skin sensitivity such as the skin thickness, the thicknesses of underlying subcutaneous fat and muscle, etc. Additionally, there may also be individual neurological differences due to sports training or other influences. For example, some martial artists learn to “block out” feelings of pain in their arms.
An individual haptic actuator may have a variable rate of vibration, duty cycle, and intensity. In one embodiment, the response is customizable for an individual user.
Thus, in some embodiments a haptic actuator situated on an arm may have a tunable degree of stimulation. Also, over the course of training a user may become more sensitive to stimulation, which might permit a reduction in the degree of stimulation required. Thus, it will be understood that while the training phase may have the same stimulation as ordinary game play, more generally the stimulation may be adjusted during a training phase based on an individual user's physiology/neurology and any training response affecting the user's sensitivity to haptic stimulation.
In one embodiment, personalization data is collected and may be stored either in a HMD or with a haptic actuator, to support personalization.
In one embodiment, a user interface, such as a dial, could be provided in the real world or as a virtual object to provide a user with personalization options.
While customization may be performed in a training phase, it will also be understood that options may be provide for a user to customize response during the play of a test game, test application, regular AR/VR game play, or regular AR/VR application.
It will also be understood that the customization may be performed based on what types of haptic actuators the user selects, the places in the body that they decide to use them, and to what extent the user desires haptic feedback. For example, some users may desire only limited haptic feedback whereas other user may desire more extensive haptic feedback.
While examples of training have been provided, it will be understood that variations are contemplated. For example, in theory different modalities may be used to train different parts of the body. For example, through injury or atrophy individual users may have different physical and neurological responses for one hand or the other. Additionally, most people generally have a preferred side (e.g., right handed or left handed). Thus, in theory the training and/or referred haptic stimulation could be selected to be different for each hand to account for individual user differences and preferences.
It will also be understood that the customization may also be performed for other reasons, such as using referred haptic feedback to provide “feelings” of size or density. For example, a rate of vibration may be varied during training to form an association with the size or density of a virtual object. Human beings have a natural feeling that tiny things vibrate more quickly than larger things. Similarly, human beings have a natural feeling that there is a difference between low density objects (e.g., Styrofoam) and high density objects (e.g., a gold bar). Thus, in one embodiment, customization and training is performed to support, for example, developing feelings of interacting differently with a small virtual object (e.g., a virtual mouse) or a large virtual object (e.g., a virtual elephant). Alternatively, the customization and training may be performed to support interacting differently with objects based on their density.
The capability to provide gloveless haptic feedback for the interactions of a human hand with virtual objects has many potential applications outside of game play. For example, in virtual sculpting tactical feedback helps an artist to shape the virtual materials they see. Conventionally, haptic gloves would be required, but these can be uncomfortable and restrictive of fine movement of the hands. Thus, the application of gloveless haptic feedback, based on referral, in accordance with embodiments of the present invention permits a user to have the benefits of haptic feedback for the interaction of their hands with virtual objects but without the disadvantages of conventional haptic feedback gloves. Similarly, there are applications in manufacturing in which there are potential advantages in cost or comfort in providing haptic feedback by referral. For example, in some manufacturing application haptic feedback may be used to provide feedback as to where parts are in space. Similarly, there are potential applications in medical training to train for feeling in the body.
It will also be understood that in one embodiment the haptic training is performed to form a mental association with specific game sensations.
For example, subtle feelings, like feelings of dread, could be trained by forming an association between a “weird” feeling in one part of the body and music or images evoking a sense of dread. As another example, a “cold” feeling could be generated in a part of a body as subtle feeling that is associated with a “zombie” or other monster. In theory, this could be done directly (e.g., via a thermoelectric cooler as the haptic actuator). However, more generally some forms of vibration/stimulation generate nervous effects similar to a cold feeling. Through training a “cold touch” feeling may be generated as a special game feature. In analogous fashion, a “hot” feeling could be referred to a different game feeling, and so on.
As another example, in a martial arts game, the virtual objects presented during training could train the user to associate a sensation, such as that generated from a haptic actuator on an arm, with feelings associated with virtual objects in the game play. For example, an association could be made the sensation of being pierced by a knife.
One of ordinary skill in the art would understand that this technique of training and using referred haptic feedback may be customized for individual virtual games or other forms of virtual entertainment.
As another application for game play, a training phase could include generating an association between a haptic stimulation on one part of the body (e.g., the arm) and proximity to a virtual or physical room boundary. One problem in virtual reality game play is that user's often begin to wander towards the walls. In one embodiment, training is performed so that a haptic actuator generates a sensation as a user approaches a wall. Through training, this can be referred to a general proprioception of where the user's body is in relation to walls. This type of referred proprioception does not have to be perfect to improve the user experience in many virtual games. It can also be performed in different ways, such as by tracking the user's movement and by activating a haptic actuation unit as the user approaches a room boundary or obstacle. A training phase may also be included to train the user to develop referred proprioception.
It will be understood that embodiments of the present invention may be used in combination with other conventional haptic feedback devices. As an example, during a training phase conventional haptic feedback gloves may be used initially and then the training perform to achieve a “transference” of skills to a gloveless approach based on referred haptic feedback.
It will be understood that embodiments of the present invention may utilize any form of haptic stimulation that can be referred from one body location to another. Without being bound by theory, it is believed that training may permit even very subtle sensations to be used. As one example, some sound frequencies are so low that they are felt, not heard. Moreover, these low frequency sound waves are capable of traveling through the body. Thus, low frequency sound generators may also be employed as haptic stimulators. Cold and heat may also be employed, such as heat/cold actuators in earbuds.
As the human brain is highly trainable to become more sensitive and aware of subtle sensations, it will thus be understood that a variety of “weak” haptic stimulation types may be utilized through training. The length of training and type of training may also be customized for the specific type of haptic stimulation.
It will be understood that embodiments of the present invention may utilize partial haptic referral of sensation. For example, if the haptic actuators are positioned on the arm, the training may result in the user receiving haptic feedback for their fingers. There may be a complete referral of sensations through training. However, it will be understood that some residual sensation may still be located at the original site of the haptic stimulation. For example, if the haptic actuators are located on the arms the user might be able to still experience some sensation there if the user focused on that sensation. However, in the case of game play, the user's attention is typically intensely focused on game play and there will be a tendency for the human brain to ignore sensations not related to the game play. Thus, it will be understood that the referral of haptic stimulation from one point to another of the body may occur with different degrees of residual sensation in the original site of the haptic stimulation. From the perspective of playing an AR or VR game, the intense focus of users on game play may permit considerable residual feeling in the original site of haptic stimulation to remain but be blocked out by the fixation of the user on game play. Similarly, in some work applications users may be so absorbed in a work task performed using AR or VR that they block out residual feelings at the original site of the haptic stimulation. That is, another aspect of haptic referral is that during normal use of a game or work task there will be a tendency for the user to ignore residual feelings at the site of the haptic stimulation.
In one embodiment the haptic actuators are clipped onto the backs of gloves or straps and refer to events supposed to happen on the fingertips. Thus, the user gets many of the benefits of haptic gloves without the discomfort and loss of mobility associated with conventional haptic gloves that require actuators in the fingertips. As one example, “fingerless” gloves could be used and the haptic actuators placed on the backs of the gloves and the haptic sensation referred to the fingertips. As another example, a wrist strap or wrist bracelet could house the haptic actuators with the referred feelings being felt in the fingertips of the user. In one embodiment, at least two haptic actuators are placed around the circumference of a user's wrist. More generally, a set of haptic actuators could be placed around the entire wrist. In this case, an individual haptic actuator may be selected at one time or two or more selected at one time to create different “patterns” of haptic stimulation around a user's wrist.
The following U.S. patents and publications are each hereby incorporated by reference:
It will be understood that exemplary methods of use and training techniques may also be embodied as computer code stored on non-transitory computer readable medium.
An illustrative embodiment has been described by way of example herein. Those skilled in the art will understand, however, that change and modifications may be made to this embodiment without departing from the true scope and spirit of the elements, products, and methods to which the embodiment is directed, which is defined by my claims.
While the invention has been described in conjunction with specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims. The present invention may be practiced without some or all of these specific details. In addition, well known features may not have been described in detail to avoid unnecessarily obscuring the invention. In accordance with the present invention, the components, process steps, and/or data structures may be implemented using various types of operating systems, programming languages, computing platforms, computer programs, and/or computing devices. In addition, those of ordinary skill in the art will recognize that devices such as hardwired devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or the like, may also be used without departing from the scope and spirit of the inventive concepts disclosed herein. The present invention may also be tangibly embodied as a set of computer instructions stored on a computer readable medium, such as a memory device.
The present application claims the benefit of U.S. Provisional Application No. 62/253,253, the contents of which are hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62253253 | Nov 2015 | US |
