Patent Application
20160239087
1. Field of the Invention
The invention relates to haptic systems, and, in particular, to a content-aware haptic system and an associated control method.
2. Description of the Related Art
In recent years, haptic technologies have emerged in academic reports and industrial products. However, current haptic technologies mostly consider how to address the haptic experience of the end user, but haptic experiences are not easy to record or to duplicate in different haptic systems. In addition, the patterns of haptic feedback are usually edited or predetermined in a specific manner, such as using a predefined mapping table to map specific symbols into corresponding haptic patterns. Conventional haptic technologies do not control the haptic feedback according to the content of an input signal such as a video signal, image signal, audio signal, etc.
Accordingly, there is demand for a haptic system and an associated control method to solve the aforementioned problems.
A detailed description is given in the following embodiments with reference to the accompanying drawings.
In an exemplary embodiment, a haptic system is provided. The haptic system includes: a processor, and control circuitry. The processor is configured to receive an input signal and perform haptic conversion on the input signal according to content of the input signal to generate a first control signal. The control circuitry is configured to generate a driving signal according to the first control signal so as to provide haptic feedback through a haptic device.
In another exemplary embodiment, a control method for a haptic system is provided. The method comprises the steps of: receiving an input signal; performing haptic conversion on the input signal according to the content of the input signal to generate a first control signal; and generating a driving signal according to the first control signal so as to provide haptic feedback through a haptic device.
The present invention can be more fully understood by reading the subsequent detailed description and examples with references made to the accompanying drawings, wherein:
The following description is of the best-contemplated mode of carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims.
The present disclosure is directed to systems and methods for providing content-aware haptic controls. Haptic systems may be used for actuation such as vibration, shape deformation (e.g., contouring a flat surface, or deformation), friction change, volumetric haptic shape, transcutaneous nerve stimulation, temperature stimulation, or other suitable actuations or combination of actuations which could provide tactile feedback to a user. Haptic systems may also be used for sensing stimuli such as, for example, contact on a display screen, patterns of contact on a screen, shape changes, physical changes of a system or component, or other suitable stimuli or combinations of stimuli which may be received. Haptic systems may sense particular stimuli, change one or more characteristics of a shape change element, or both. Haptic systems may perform sensing functions and actuating functions by a integrated haptic device. In some embodiments, haptic systems may be coupled to a display screen, audio processing circuitry, image processing circuitry, device hardware, or other system to provide for any combination of tactile, visual, and audio interactions. Actuation may occur, in some embodiments, substantially normal to a substantially planar surface, which may allow for three dimensional contouring of the planar surface.
The control circuitry 120 is configured to receive the control signal 20 and generate a driving signal 30 so as to control the haptic device 130 to provide haptic feedback. For example, the control circuitry 120 may comprise a digital-to-analog converter (DAC) 121 that is configured to convert a digital control signal 20 received from the processing unit 110 into the driving signal 30, e.g. an audio waveforms. The haptic feedback provided by the haptic device 130 may include vibration, shape deformation, friction change, volumetric haptic shape, transcutaneous nerve stimulation, temperature stimulation etc., but the invention is not limited thereto.
In an embodiment, given that the input signal 10 is a video signal and the haptic device 130 including vibrator to provide vibration haptic feedback, the processing unit 110 may analyze the video signal to obtain the motion vectors of macroblocks of an image frame in the video signal. The magnitude and/or frequency of the driving signal 30 depend on the motion vectors of the macroblocks of the image frame. For example, when the scene of the video signal moves fast, it indicates that the magnitude of the motion vectors of the image frame in the video signal would be larger, and thus the processing unit 110 may provide a control signal 20 indicating a higher amplitude and/or frequency of vibration to the control circuitry 120, so that the control circuitry 120 may provide a driving signal 30 to control the haptic device 130 to provide vibration haptic feedback having a corresponding amplitude and/or frequency. It should be noted that the control circuitry 120 may provide a driving signal 30 to control the haptic device 130 so as to provide a global vibration haptic feedback for the whole haptic device 130. In yet another embodiment, the control circuitry 120 may provide a plurality of driving signals 30 to locally control a plurality of vibrators located in specific regions of the haptic device 130 so as to provide regional vibration haptic feedbacks respectively.
In addition, the processing unit 110 may analyze the video signal to obtain the foreground object, its position, and motion vectors of the foreground object. The processing unit 110 may generate the control signal 20 to control at least a local vibrator of the haptic device 130 on the position at which the foreground object is presented. In some embodiments, the processing unit 110 may also detect the content change of the video signal, and generate the control signal 20 corresponding to the content change.
Moreover, the processing unit 110 may also analyze the acoustic content of the video signal, and may generate the control signal 20 based on the acoustic content: For example, when there is a huge explosion scene in the video signal, and the associated acoustic content may have a sudden explosion sound with a high magnitude. Accordingly, the control signal 20 generated by the processing unit 110 may indicate a sudden strong vibration on the haptic device 130 while the explosion occurs.
In another embodiment, the haptic device 130 includes electrostatic generator which is capable of providing electrostatic haptic feedback. For example, the haptic device 130 may be an electrostatic passive haptic display or an electrostatic passive haptic touchpad and the input signal 10 is a video signal, but the invention is not limited thereto. Given that the haptic device 130 is an electrostatic passive haptic display, the user may sense friction change while sliding his/her finger or fingers on the surface of the haptic device 130, and the friction force sensed by the user corresponds to the content of the image. Additionally, the friction force (i.e. sensing slippery or rough) sensed by the user can be controlled by the driving signal 30 applied to the haptic device 130. In some embodiments, given that the haptic device 130 is an electrostatic passive haptic display, the user may place his/her finger or fingers directly on the surface of the haptic device 130, and the voltage difference occurs between the finger (as an electrode) and the haptic device 130. This allows a direct touch without the use of any additional components. In addition, the surface of the haptic device 130 may include several segmented electrodes placed in a plurality of different regions. In some other embodiments, given that the haptic device 130 is an electrostatic passive haptic touchpad, the user may experience tactile feedback by putting his/her finger or fingers on the touchpad and explores the surface of the haptic device 130. For example, there may be a cursor or pointer on a display of the haptic system 100, and the user may sense the friction haptic feedback where the cursor or pointer is located when putting his finger on the surface of the haptic device 130.
In the aforementioned embodiments, the processing unit 110 may perform a two-dimensional frequency-domain transformation (e.g. digital wavelet transform (DWT) or digital cosine transform (DCT)) on the image frame of the video signal, and thus a map in the frequency domain can be obtained. When pixel values in a specific image block vary a lot, it indicates that the frequency in the specific image block is higher, and thus the control signal 20 generated by the processing unit 110 may control the control circuitry 120 to provide the driving signal 30 with higher voltage or frequency, so that the user may sense that the friction force on the specific image block is larger (i.e. sensing a rough surface) while sliding on the corresponding specific image block of the haptic device 130 via his finger.
In some embodiments, the haptic device 130 (e.g. an electrostatic passive haptic display) could be able to provide friction haptic feedback. The applied driving signal 30 to drive the haptic device 130 for generating a friction haptic feedback depends on the magnitude of motion vectors of the image frame. Additionally, the control circuitry 120 may provide a plurality of driving signals 30 to control different segmented electrodes in the haptic device 130 to provide a friction haptic feedback at which the foreground object or an obvious image feature (e.g. a human face) is appeared in the image frame. Alternatively, the friction force provided by the haptic device 130 may also depend on the magnitude or frequency of the acoustic content of the incoming video signal, and the details can be found in the aforementioned embodiment.
Similarly, given that the haptic device 130 is further capable of providing deformation haptic feedback (e.g. through a display screen having a deformation function on its surface), the processing unit 110 may also generate the control signal 20 associated with the content of the video signal. For example, as described above, the degree of deformation haptic feedback also corresponds to the magnitude of the motion vectors and/or the content change in the video signal.
In an embodiment, the haptic device 130 is a haptic deformation display having a plurality of deformation components, e.g. micro-fluidics display panel, and the processing unit 110 is capable of converting an incoming stereoscopic video signal or 3D image into a control signal 20 indicating deformation magnitude and shape of the deformation components of different regions on the haptic device 130. It should be noted that the surface of the haptic device 130 can be divided into a plurality of regions, and the control circuitry 120 may generate a plurality of driving signals 30 to control the deformation component of respective region according to the control signal 20 from the processing unit 110. For example, the haptic deformation display is originally in a first-shape configuration, and the deformation component may at least partially define the shape of the haptic deformation display, thereby causing the shape of the haptic deformation display to deform into a second-shape configuration that is different from the first-shape configuration. The second-shape configuration may substantially be maintained.
The processing unit 110 may calculate depth information (e.g. a depth map) associated with image frame in the stereoscopic video signal or 3D image, and the deformation height of the deformation component of a specific region corresponds to the associated depth information. In addition, the processing unit 110 may perform foreground segmentation on the incoming video signal or image to obtain a foreground object. The control signal 20 controls the individual deformation component of each region of the haptic device 130 to generate the deformation shape corresponding to the segmentation result of the foreground object.
In some embodiments, control signal 20 is generated according to the temporal motion prediction results of the incoming stereoscopic video signal or 3D image when the haptic system 100 is used in a video codec system. Similarly, the haptic feedback (e.g. slippery or rough) may also correspond to the magnitude, or correspond to motion vectors in the stereoscopic video signal or 3D image. In addition, the processing unit 110 may perform foreground segmentation on the stereoscopic video signal or 3D image to obtain a foreground object so as to generate the control signal 20 accordingly by the processing unit 110. The control circuitry 120 may generate the driving signals 30 to control the corresponding deformation components where the foreground object is presented on the haptic deformation display. Furthermore, the deformation behavior may also correspond to the acoustic content in the stereoscopic video signal, and the aforementioned embodiments can be referred to for the details.
In an embodiment, the haptic device 130 equipped with transcutaneous electrical nerve stimulation device having a plurality of electrodes. The processing unit 110 is capable of converting an incoming video signal or audio signal into a control signal 20. The control circuitry 120 may provide a plurality of driving signals 30 to control the plurality of electrodes in the haptic device 130 to provide a transcutaneous nerve stimulation haptic feedback. The driving signals 30 indicate amplitude, frequency and/or duration of electrical impulses which would be applied by the plurality of electrodes on the haptic device 130.
In yet an embodiment, the haptic device 130 equipped with heaters. The processing unit 110 is capable of converting an incoming video signal or audio signal into a control signal 20. The control circuitry 120 may provide a plurality of driving signals 30 to control the heaters in the haptic device 130 to provide thermal stimulation haptic feedback. The driving signals 30 control thermal energy, and/or heating duration of the heaters of the haptic device 130.
In view of the above, the haptic system 100 is able to convert the input content into various types of haptic feedback, e.g. vibration, shape deformation, friction change, volumetric haptic shape, transcutaneous nerve stimulation, and/or temperature stimulation mentioned in the embodiments in above.
In yet another embodiment, the input content can be processed so as to generate driving signal to provide various haptic feedback with the help of sensor data. For example, the haptic system 100 may further includes at least a sensor 140 such as GPS sensor, ambient light sensor, proximity sensor, thermal sensor, accelerometer, pressure sensor, heart-rate sensor, or UV sensor, etc. The sensor 140 provides sensor data 40 to the processing unit 110. In an embodiment, the processing unit 110 receives the input video signal 10 and GPS data from the GPS sensor and generates a control signal 20 according to the input video signal 10 and GPS data, where the control signal 20 may be converted to the driving signal 30 to control the haptic device 130 to provide the haptic feedback, e.g. vibration haptic feedback. For example, the vibration magnitude may depend on the motion vectors of the image frame of the video signal, and also depend on the GPS data e.g. geographical location of the user such as at home or office. The vibration haptic feedback may be weaker when the user is at home, and the vibration haptic feedback may be stronger when the user is in the office. Additionally, the processing unit 110 may also generate the control signal 20 according to the video signal and a plurality of sensor data 40 so as to control the haptic feedback of the haptic device 130.
In an embodiment, the haptic device 130 further comprises an ultrasound haptic component. The ultrasound haptic component is capable of producing a volumetric haptic shape to the user by vibrating the air particles near the ultrasound haptic component, so that the user may sense the friction haptic feedback by “touching” the mid-air where the ultrasound haptic component renders the volumetric haptic shape. In addition, the ultrasound haptic component is capable of detecting the touch position where the user is touching in mid-air, and sends the touch position information 50 to the processing unit 110. Thus, the processing unit 110 may enhance the ultrasound haptic feedback on the touch position.
In view of the above, the haptic system 100 is able to convert the input content and the sensor data 40 into various types of vibration, shape deformation, friction change, and/or volumetric haptic shape mentioned in the embodiments in above.
In another embodiment, the sensors 140 of the haptic system 100 may further comprise a microphone for detecting the acoustic signal performed by the user. And the sensors 140 of the haptic system 100 may further comprise a camera for detecting the motions performed by the user. For example, the processing unit 110 can be further controlled by the user through a gesture, a voice command, an ultrasound signal, or a pose captured by the microphone or the camera. Specifically, when the processing unit 110 detects the gesture, voice command, ultrasound signal, or pose performed by the user, the processing unit 110 may apply a specific haptic signal instead of the original control signal, so that the control circuitry 120 would generate the driving signal 30 according to the specific haptic signal. It should be noted that the haptic signal includes vibration, friction, or deformation, but the invention is not limited thereto.
In an embodiment, the haptic system 100 may further comprise a network interface unit 150 configured to communicate with other devices using wired or wireless protocols such as LAN, Bluetooth, Wifi, LTE etc. For example, the processing unit 110 of the haptic system 100 may receive a status signal 60 indicating a remote user's action or a remote object's status from a remote electronic device (not shown in
In yet another embodiment, the processing unit 110 may analyze a input image signal and identify objects having different materials from the image signal by using object recognition techniques or material data base from an external source. And the processing unit 110 is configured to be capable of generating haptic signals corresponding to different materials such as cloth, glass, paper, metal, air, but the invention is not limited thereto. For example, given that the haptic device 130 provides friction haptic feedback, the user may sense that the cloth may be rough and the metal may be slippery when touching on the associated portion on the haptic device 130 e.g. a electrostatic passive haptic display. That is, different material has a corresponding haptic signal that causes an individual haptic feedback, e.g. different friction haptic feedback driven by driving signals 30 having different magnitude and/or frequency.
Furthermore, the haptic signals of different materials can be recorded using analog signals such as audio signals. Specifically, each material has its own acoustic material haptic file. When the processing unit 110 recognizes the materials in the video signal, the processing unit 110 may retrieve the acoustic material haptic files of the recognized materials from a database (not shown in
In an embodiment, the input signal 10 may include an analog content, e.g. an analog audio signal, which can be regarded as the control signal 20. Thus, the analog content could be treated as the driving signal 30 and be sent to the haptic device 130 for proving haptic feedback directly. For example, the processing unit 110 may determine whether the input signal 10 is an analog signal. If the input signal 10 is an analog signal, the processing unit 110 may directly send the analog signal to the haptic device 130. For example, the direct output haptic feedback from the input audio signal can also reflect the vibration from the air wave such as collision between materials, or the explosion of fireworks.
If the input signal 10 is a digital signal, the processing unit 110 may perform haptic conversion to convert the digital signal into an analog control signal that is transmitted to the control circuitry 120. However, both the digital content and the analog content may exist in a multimedia file. For example, when the input signal is a video signal including video images and an analog acoustic signal, the processing unit 110 may perform haptic conversion on the video signal to generate a first control signal, and directly send the analog acoustic signal to the control circuitry 120 as the second control signal. Accordingly, the haptic device 130 performs haptic feedback according to the first control signal and the second control signal.
While the invention has been described by way of example and in terms of the preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements (as would be apparent to those skilled in the art). Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.
This application claims the benefit of U.S. Provisional Application No. 62/116,710 filed on Feb. 16, 2015, the entirety of which is incorporated by reference herein.
| Number | Date | Country | |
|---|---|---|---|
| 62116710 | Feb 2015 | US |
