Patent Application
20210365122
The following embodiments relate to a haptic data generating technology and a haptic effect providing technology.
Game developers use visual and auditory effects to provide an immersive game play. For example, when a user plays a game on a smartphone, a visual effect for a game event is provided through a screen or an auditory effect for the game event is provided through a speaker or earphone. In addition, in recent years, a vibration effect is sometimes used to increase the sense of experience for the game event. As prior art, Korean Patent Publication No. 1998-0032560 ‘Operating Device and Game Device for Game Machines’ discloses an operating device for a game machine including a response member operated to perform feedback to a user in the form of vibration, sound, light, or a combination thereof.
A device for generating haptic data linked with a game program according to an embodiment may include: a controller generating haptic data for an event occurring during game play based on haptic data setting information; and a communication unit transmitting the generated haptic data to a haptic effect providing device.
The event may include at least one of a game content event occurring on a game content of the game program and a user input event for a user input input for the game play.
The haptic data setting information may include setting information for at least one of an application range of a haptic effect, a haptic pattern, and a haptic strength.
When one or more content events selected by a user occur, the communication unit may transmit haptic data corresponding to the game content event which occurs.
The controller may perform the fast Fourier transform processing for the audio data corresponding to each event, select a target frequency band of the audio data based on the result of performing the fast Fourier transform processing, perform filtering for the selected target frequency band by using the bandpass filter, perform the inverse fast Fourier transform processing for the audio data in the target frequency band in which the filtering is performed, and determine the haptic pattern corresponding to the event based on the threshold in the result of performing the inverse fast Fourier transform processing.
The controller may perform the fast Fourier transform processing for the audio data corresponding to the event, extract a frequency in which a result value of performing the fast Fourier transform processing is largest, and generate the haptic data based on the haptic strength corresponding to the extracted frequency.
The haptic data may include a first element indicating the type of message, a second element indicating a size of a message value, and a third element including message information.
The controller may generate haptic data to be provided to a plurality of users which performs the game play together.
The device for generating haptic data according to an embodiment may further include a USB host transmitting, when the haptic data generating device is connected to the haptic effect providing device by wire through a cable, power and the haptic data to the haptic effect providing device through the cable.
A device for providing a haptic effect according to an embodiment may include: a communication unit receiving haptic data from a haptic data generating device; a haptic effect generating unit generating the haptic effect under the control of a controller; and the controller controlling the generation of the haptic effect based on the received haptic data, in which the haptic data may be associated with an event which occurs during game play and generated based on haptic data setting information.
The controller may generate haptic pattern data using information on a haptic pattern index and a haptic strength included in the haptic data and control the generation of the haptic effect based on the haptic pattern data.
The controller may generate the haptic pattern data based on a haptic pattern duration corresponding to the haptic pattern index.
The device for providing a haptic effect according to an embodiment may further include a lighting effect reproducing unit generating a lighting effect under the control of the controller, in which the controller may control the generation of the lighting effect based on the haptic data.
The controller may extract information on the haptic strength from the haptic data and determine at least one of a lighting color and a lighting strength based on the information on the haptic strength.
A method for generating haptic data linked with a game program according to an embodiment may include: generating haptic data for an event occurring during game play based on haptic data setting information; and transmitting the generated haptic data to a haptic effect providing device.
A method for providing a haptic effect according to an embodiment may include: receiving, from a haptic data generating device, haptic data associated with an event occurring during game play; generating haptic pattern data based on the haptic data; and generating a haptic effect based on the haptic pattern data, in which the haptic data may be generated based on haptic data setting information.
According to an embodiment, it is possible to provide a haptic effect and/or a lighting effect linked to a game program through a wearable device, a game controller, or a mobile device accessory device when a user plays a game.
According to an embodiment, it is possible to provide the haptic effect and/or the lighting effect simultaneously to various users who are playing the same game.
Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, various changes may be made to the embodiments, and thus the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes to the embodiments are included in the scope of the rights.
The terms used in the examples are used for illustrative purposes only and should not be interpreted as limiting. A singular form may include a plural form if there is no clearly opposite meaning in the context. In the present application, it should be understood that term “include” or “have” indicates that a feature, a number, a step, an operation, a component, a part or the combination thereof described in the specification is present, but does not exclude a possibility of presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof, in advance.
If not contrarily defined, all terms used herein including technological or scientific terms have the same meanings as those generally understood by those skilled in the art. Terms which are defined in a generally used dictionary should be interpreted to have the same meaning as the meaning in the context of the related art, and are not interpreted as an ideal meaning or excessively formal meanings unless clearly defined in the present application.
In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiment, a detailed description of related known technologies will be omitted if it is determined that they make the gist of the embodiment unclear.
A haptic effect providing system 100 provides a haptic effect and/or a lighting effect to a user when the user plays a game program in link with the game program. Therefore, the user may feel a higher sense of immersion and fun. For example, the haptic effect providing system 100 may provide a haptic effect to the user according to the timing at which a game character shoots a gun or the game character is struck in game play, through which the user may receive a feel as if actually shooting the gun or striking is made. As another example, the haptic effect providing system 100 may generate haptic pattern data corresponding to a sound pattern of audio data by analyzing audio data output from a game and provide the haptic effect to the user based on the generated haptic pattern data.
Referring to
The haptic effect providing device 140 is a device that provides a haptic effect (or haptic stimulus) and/or a lighting effect to the user based on the haptic data transferred from the haptic data generating device 110. The haptic effect providing device 140 may also be referred to as a ‘haptic device’. The haptic effect providing device 140 may play back the haptic data and express a haptic feedback such as vibration or tapping and provide a haptic stimulus having a specific pattern to the user. The haptic effect providing device 140 may have any one form of a wearable device, a mobile device accessory device, a game console controller, a game pad, a keyboard, a mouse, and a joystick, for example, but the form is not limited thereto. The haptic effect providing device 140 includes a device linked with the device in which the game program is driven.
In an embodiment, when an event occurs on the game play, the haptic data generating device 110 may generate the haptic data corresponding to the game content and transmit the generated haptic data to the haptic effect providing device 140. The haptic effect providing device 140 may play the haptic data received from the haptic data generating device 110 to provide the haptic effect to a user, and additionally provide the lighting effect and a sound effect related to the game contents to the user. The haptic effect providing device 140 may comprehensively provide the haptic effect, the lighting effect, and the sound effect for the event to the user. The user may receive the haptic effect and/or the lighting effect through a wearable device, the keyboard, or the mouse while playing the game through the personal computer or the smartphone. In this case, the user is provided with not a haptic effect of a simple pattern, but a haptic effect of various patterns linked to the game content. In some embodiments, the haptic effect providing system 100 may provide the haptic effect and/or the lighting effect simultaneously to various users who play the same game.
In an example, when the user plays the game alone or several users play the game together, it is assumed that the user playing the game wears the wearable device in the form of a watch or a band. When an event such as gunshot or kick occurs in the game program, haptic pattern data corresponding to the event is played in the wearable device to generate the vibration, and a lighting effect reproducing unit (e.g., LED) provided in the wearable device may be driven. Therefore, the game play immersion and enjoyment by the user may be enhanced. Further, since the user may feel the haptic effect while hearing audio related to the game content, an effect of the game may be maximized. Further, it is possible to effectively transfer the game effect on the game content to a user who may not hear the audio, such as a hearing impaired user through the haptic effect or the lighting effect.
As illustrated in
The haptic data generating device 110 includes a controller 125 and a communication unit 130. In some embodiments, the haptic data generating device 110 may further include an input interface 115 and an output interface 120. The input interface 115 receives a user input. For example, the input interface 115 may receive touch data of the user through a touch screen display and transmit the received touch data to the controller 125. The output interface 120 may play audio data related to the game content and output audio through a speaker or earphone.
The controller 125 may control an operation of the haptic data generating device 110 and include one or more processors. In some embodiments, the controller 125 may control the game program. In this case, the controller 125 controls the start and end of the game program, communication linkage with the haptic effect providing device 140, and event processing.
The controller 125 generates the haptic data for the event that occurs during the game play. Here, the event includes one or more of a game content event occurring on the game content of the game program and a user input event for a user input input for the game play. The game content event includes, for example, a case where a game character is damaged or a specific game object appears on the game content. The user input event includes, for example, a case where the user applies a touch input on the touch screen display to manipulate the game character.
The game content event may be an event which occurs by the user input event. For example, the game character is manipulated by the user input event such as the touch input of the user and when a change in game content thus occurs (e.g., a specific game object is detonated), the change in game content corresponds to the game content event. In some embodiments, the game content event may also include an event such as an alarm for a specific situation on the game content, such as an alarm for a dangerous moment of the game character. Such a haptic effect applied to various events may help the user to better play the game, and accordingly, the user may further enhance a game play ability.
In an embodiment, the controller 125 may generate the haptic data based on haptic data setting information. The haptic data setting information may include setting information for at least one of an application range of the haptic effect, the haptic pattern, and a haptic strength which may be determined by the user input. The user may directly set the application range of the haptic effect, the haptic pattern, and/or the haptic strength on the game program, or the haptic data may be set according to setting information designated by a developer of the game program. The user may select one or more game content events to which the haptic effect is to be applied and also set the haptic strength to be applied to each game content event or user input event. In this case, the controller 125 may generate the haptic data based on the haptic strength set by the user in some embodiments.
In an embodiment, the controller 125 may generate the haptic data based on the audio data corresponding to each event. The controller 125 processes the haptic pattern for each period set at the time when audio data is transmitted. As a more specific example, the controller 125 may perform fast Fourier transform (FFT) processing on the audio data of the game program and select a target frequency band of the audio data based on a result of performing the FFT processing. For example, the controller 125 may select a frequency band including a frequency in which a value in which the FFT processing is performed is largest as the target frequency band. Thereafter, the controller 125 may perform filtering for the target frequency band by using a bandpass filter and perform inverse fast Fourier transform (IFFT) processing for the audio data in the target frequency band in which the filtering is performed. The controller 125 may determine the haptic pattern based on a threshold in the result of performing the IFFT processing. For example, the controller 125 may convert data of which filtering is performed into a time domain through the IFFT processing, extract data values of the threshold or more from the data converted into the time domain, and then, add the extracted data values in units of a specific time interval. The controller 125 may determine the haptic pattern corresponding to the event by determining that the haptic effect occurs when a result of adding the data values satisfies a specific condition. Further, the controller 125 may extract a frequency in which the result value of performing the FFT processing for the audio data corresponding to the event of the game program is largest and generate the haptic data based on the haptic strength corresponding to the extracted frequency. The haptic strength corresponding to a magnitude of each frequency may be defined in advance.
The communication unit 130 transmits the haptic data to the haptic effect providing device 140 through the network 170. For example, when receiving the haptic data from the controller 125, the communication unit 130 may transmit the haptic data by using Bluetooth, Bluetooth Low Energy (BLE), Wi-Fi, and Long Term Evolution (LTE). When, for example, the game content event or the user input event occurs, the communication unit 130 may transmit the haptic data corresponding to each event to the haptic effect providing device 140. Alternatively, when one or more game content events selected by the user in the haptic data setting information occur, the communication unit 130 may transmit the haptic data corresponding to the game content event which occurs to the haptic effect providing device 140.
The haptic effect providing device 140 includes a communication unit 145, a controller 150, and a haptic effect reproducing unit 155. In some embodiments, the haptic effect providing device 140 may further include a lighting effect reproducing unit 160.
The communication unit 145 receives the haptic data to the haptic data generating device 110 through the network 170. The haptic data may be associated with the event which occurs during the game play and may be generated based on the haptic data setting information. The haptic data may include data regarding the haptic pattern.
The controller 150 may control the operation of the haptic effect providing device 140 and include one or more processors. The controller 150 controls the occurrence of the haptic effect based on the received haptic data. The controller 150 may generate haptic pattern data using information on a haptic pattern index and a haptic strength included in the haptic data, and control the occurrence of the haptic effect based on the haptic pattern data. The controller 150 may generate the haptic pattern data based on a haptic pattern duration corresponding to the haptic pattern index.
The haptic effect reproducing unit 155 reproduces the haptic pattern data under the control of the controller 150 and generates the haptic effect. The haptic effect reproducing unit 155 may include one or more actuators that generate the haptic stimulus.
The lighting effect reproducing unit 160 generates the lighting effect under the control of the controller 150. For example, the lighting effect reproducing unit 160 may include a lighting device such as an LED. The controller 150 may control the occurrence of the lighting effect based on the haptic data. For example, the controller 150 may extract information on the haptic strength from the haptic data and determine at least one of a lighting color and a lighting strength based on the extracted haptic strength information. For example, the controller 150 may display a red lighting more brightly as the haptic strength is stronger, and may display a blue lighting brighter as the haptic strength is weaker.
Referring to
When the haptic data generating device 210 is connected to the haptic effect providing device 240 by wire through the cable, the USB host 225 may transmit power and haptic data to the haptic effect providing device 240 through the cable. In this case, the haptic effect providing device 240 may simultaneously receive power which may be used for driving the actuator together with the haptic data from the haptic data generating device 210. In one embodiment, the haptic data generating device 210 may be included in the smartphone and driven, and in this case, the haptic effect providing device 240 may be a haptic patch for a game that may be attached to the smartphone. At this time, the USB host 225 may be, for example, a charging jack interface, and when the haptic data generating device 210 and the haptic effect providing device 240 are connected by wire, the power may be supplied from a charging portion of the smartphone to the haptic effect providing device 240 and at the same time, the haptic data may be transferred.
Referring to
The user may determine whether to set the application range of the haptic effect through the setting interface (310). The application range of the haptic effect indicates a haptic effect processing range for the event. When the application range is not set (i.e., not set), the application range of the haptic effect is set to a range designated by a developer (game program developer) (315). In contrast, when the user sets the application range, the application range of the haptic effect is set to the range set by the user (320). In an embodiment, in the basic setting on the game program, the application range of the haptic effect may be set to the range designated by the developer. The user may determine whether to generate the haptic effect for each event through the setting interface on the game program. The game program may generate the haptic effect to the application range determined by the user.
Furthermore, the user may determine whether to set the haptic pattern through the setting interface (325). When the user does not set the haptic pattern, the haptic pattern is set to a haptic pattern designated by the developer (330). The user may select the haptic pattern for each event and in this case, a haptic pattern corresponding to the corresponding event is set for the event selected by the user (335). For an event not selected by the user, the haptic pattern may be set to the haptic pattern provided by the developer.
When the user does not set the haptic pattern for a specific event, the haptic pattern designated by the developer may be determined based on the audio data corresponding to the event. For example, the haptic pattern may be determined through a series of processing processes such as Fast Fourier Transformation (FFT) processing, selection of a target frequency band, bandpass filtering processing, and haptic pattern extraction using a threshold for the audio data. This will be described in more detail in
Furthermore, the user may determine whether to set the haptic strength for each event through the setting interface (340). When the user does not set the haptic strength, the haptic strength is set to the haptic strength which the developer designates for each haptic effect. The user may select the haptic strength of the haptic effect for each event and in this case, a haptic strength corresponding to the corresponding event is set for the event selected by the user (350). For an event not selected by the user, the haptic strength may be set to a predefined haptic strength.
When the user does not set the haptic strength for a specific event, the haptic data generating device may perform FFT processing for audio data (e.g., sound effect) corresponding to the corresponding event (345). The haptic data generating device may extract a maximum frequency (e.g., a frequency in which dB is largest) having a largest value from an FFT processing result (355). The haptic data generating device may determine the haptic strength corresponding to the event based on the extracted maximum frequency (360). The haptic data generating device may determine the haptic strength corresponding to the maximum frequency as the haptic strength corresponding to the corresponding event. The haptic strength corresponding for each frequency may be predefined as shown in Table 1 below, for example.
For example, as a result of FFT processing of audio data corresponding to event A, assuming that the maximum frequency with the largest FFT conversion value is 240 Hz, the haptic strength corresponding to the event A may be determined as haptic strength ‘4’ in a frequency range including a frequency of 240 Hz.
When the user selects the haptic strength, the haptic strength may be determined for each event as shown in Table 2 below, with a martial art game as an example. A duration which the developer designates for each event is determined as a duration of the haptic pattern.
According to such a process, when at least one of the application range of the haptic effect, the haptic pattern, and the haptic strength is set, the set contents may be stored in haptic data setting information (370). After the haptic data setting information is stored (370), the haptic data is generated based on the haptic data setting information after the game play starts. According to the above embodiment, it is possible to provide the haptic effect by processing the audio data corresponding to the event without setting the haptic effect separately for each event.
Referring to
In step 420, the haptic data generating device may select the target frequency band of the audio data based on the result of performing the FFT processing in step 410. In an embodiment, the haptic data generating device may split the audio data converted into the frequency domain for each frequency band and select a frequency band including a frequency in which a value subjected to the FFT processing is largest as the target frequency band. In step 430, the haptic data generating device may perform filtering for the target frequency band by using a bandpass filter.
In step 440, the haptic data generating device may perform inverse fast Fourier transform (IFFT) processing for the audio data in a target frequency band in which filtering in step 430 is performed. Through the IFFT processing, data may be converted into the time domain from the frequency domain.
In step 450, the haptic data generating device may perform threshold and addition processing in the result of performing the IFFT processing in step 440. The haptic data generating device may extract data values of a threshold or more from the data converted into the time domain and then, add the extracted data values in units of specific time interval. In step 460, the haptic data generating device may determine the haptic pattern based on a threshold and addition processing result in step 450. The haptic data generating device may determine the haptic pattern by determining that the haptic effect occurs when an addition result of the data values determined in step 450 satisfies a specific condition.
As such, it is possible to provide the haptic effect by processing the audio data for each event without setting the haptic effect separately for each event.
According to an embodiment, the haptic data may include information on the haptic pattern index and the haptic strength and may include various information as shown in Table 3 below.
Meanwhile, the haptic data may be constructed in a message format illustrated in
Referring to
In step 640, the haptic data generating device may generate the haptic data for the event which occurs during the game play based on the haptic data setting information. In an embodiment, the haptic data generating device may generate the haptic data based on the haptic data setting information including setting information for at least one of the application range of the haptic effect, the haptic pattern, and the haptic strength. The haptic data setting information may be determined by the user input and the user may select the application range of the haptic effect, the haptic pattern, and the haptic strength through the setting interface in the game program.
Even if not set by the user, the haptic data generating device may perform the fast Fourier transform processing for the audio data corresponding to each event, select a target frequency band of the audio data based on the result of performing the fast Fourier transform processing, perform filtering for the target frequency band by using the bandpass filter, perform the inverse fast Fourier transform processing for the audio data in the target frequency band in which the filtering is performed, and determine the haptic pattern based on the threshold in the result of performing the inverse fast Fourier transform processing. Furthermore, the haptic data generating device may extract a frequency in which the result value of performing the FFT processing for the audio data is largest and generate the haptic data corresponding to the event based on the haptic strength corresponding to the extracted frequency. The contents described through
In step 650, the haptic data generating device may transmit, to the haptic effect providing device, the haptic data generated in step 640.
Referring to
In step 740, the haptic effect providing device provides the haptic effect to the user by generating the haptic effect based on the haptic pattern data. The haptic effect providing device may drive the actuator based on the haptic pattern data and a time at which each haptic pattern is provided may be processed according to the duration for each event corresponding to the haptic strength. That is, the duration may be determined according to the haptic pattern index (which is the same as the index for the event).
In some embodiments, in step 750, the haptic effect providing device may provide the lighting effect to the user by generating the lighting effect based on the haptic pattern data. The haptic effect providing device may generate the lighting effect based on the information on the haptic strength extracted from the haptic pattern data. For example, the haptic effect providing device may determine a color or a brightness of LED lighting by using the information on the haptic pattern index and the haptic strength included in the haptic pattern data and generate the lighting effect while the haptic effect is provided.
The contents described through
The method according to the embodiment may be implemented in a form of a program command which may be performed through various computer means and recorded in the computer readable medium. The computer readable medium may singly or combinationally include a program command, a data file, or a data structure or a combination thereof. The program command recorded in the medium may be specially designed and configured for the embodiment, or may be publicly known to and used by those skilled in the computer software field. Examples of the computer-readable recording medium include magnetic media such as a hard disk, a floppy disk, and a magnetic tape, optical media such as a CD-ROM and a DVD, magneto-optical media such as a floptical disk, and a hardware device which is specifically configured to store and execute the program command such as a ROM, a RAM, and a flash memory. Examples of the program command include a high-level language code executable by a computer by using an interpreter, and the like, as well as a machine language code created by a compiler. The hardware device may be configured to be operated with one or more software modules in order to perform the operation of the embodiment and vice versa.
The software may include a computer program, code, instructions, or a combination of one or more thereof, and configure the processing unit to operate as desired, or instruct a processing device independently or collectively. Software and/or data may be interpreted by the processing device or may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or a transmitted signal wave in order to provide instructions or data to the processing device. The software may be distributed on a computer system connected through the network and stored or executed by a distributed method. The software and the data may be stored in one or more computer readable recording media.
As described above, although the embodiments have been described by the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc., described are collected or combined in a form different from the described method, or even if the components are replaced or substituted by other components or an equivalent, an appropriate result can be achieved.
Therefore, other implementations, other embodiments and claims and equivalents fall within the scope of the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2018-0142103 | Nov 2018 | KR | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/KR2019/015010 | 11/7/2019 | WO | 00 |
