The present disclosure relates to processing of the audio signal. More particularly, the present disclosure relates to a system with sound adjustment capability, a method of adjusting sound and a non-transitory computer readable storage medium.
Virtual reality (VR) is a technology of using a computer to simulate a three-dimensional virtual world providing the user with visual, auditory, tactile and other sensory simulations. Headphones are commonly incorporated in VR devices to provide immersive binaural audio effects. However, not only sounds of the real world are blocked by the headphone, but also other people cannot hear sounds the headphone provided to the user, which makes the communication between the user and the user's colleagues or teammates become difficult.
The disclosure provides a system with sound adjustment capability. The system includes a head-mounted device, a first loudspeaker and at least one processor. The first loudspeaker is detachable from the head-mounted device. The at least one processor is configured to detect a plurality of positions and a plurality of orientations of the head-mounted device and the first loudspeaker to determine whether the first loudspeaker is detached from the head-mounted device. The at least one processor is further configured to modify a first audio signal by at least one first filter or at least one second filter to generate a filtered first audio signal. The at least one processor uses the at least one first filter in response to that the first loudspeaker is coupled to the head-mounted device, and uses the at least one second filter in response to that the first loudspeaker is detached from the head-mounted device. The filtered first audio signal is configured to be transmitted to the first loudspeaker to drive the first loudspeaker.
The disclosure provides a method of adjusting sound. The method is applicable to a system including a head-mounted device and a first loudspeaker detachable from the head-mounted device, and includes the following operations: detecting a plurality of positions and a plurality of orientations of the head-mounted device and the first loudspeaker to determine whether the first loudspeaker is detached from the head-mounted device; modifying a first audio signal by at least one first filter or at least one second filter to generate a filtered first audio signal, in which the at least one first filter is used in response to that the first loudspeaker is coupled to the head-mounted device, and the at least one second filter is used in response to that the first loudspeaker is detached from the head-mounted device; and transmitting the filtered first audio signal to the first loudspeaker to drive the first loudspeaker.
The disclosure provides a non-transitory computer readable storage medium storing a plurality of computer readable instructions for controlling a system including at least one processor, a head-mounted device and a first loudspeaker detachable from the head-mounted device. The plurality of computer readable instructions, when being executed by the at least one processor, cause the at least one processor to perform: detecting a plurality of positions and a plurality of orientations of the head-mounted device and the first loudspeaker to determine whether the first loudspeaker is detached from the head-mounted device; modifying a first audio signal by at least one first filter or at least one second filter to generate a filtered first audio signal, in which the at least one first filter is used in response to that the first loudspeaker is coupled to the head-mounted device, and the at least one second filter is used in response to that the first loudspeaker is detached from the head-mounted device; and transmitting the filtered first audio signal to the first loudspeaker to drive the first loudspeaker.
It is to be understood that both the foregoing general description and the following detailed description are by examples, and are intended to provide further explanation of the disclosure as claimed.
Reference will now be made in detail to the present embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.
The first loudspeaker 120A and the second loudspeaker 120B are coupled to the head-mounted device 110 on opposite first and second terminals 114 and 116 of the head-mounted device 110, respectively, and are detachable from the head-mounted device 110. In the situation that the first loudspeaker 120A and the second loudspeaker 120B are coupled to the head-mounted device 110, the first loudspeaker 120A and the second loudspeaker 120B are configured to be positioned at locations corresponding to entrances of a user's left and right ear canals. On the other hand, when the first loudspeaker 120A and the second loudspeaker 120B are detached from the head-mounted device 110, the first loudspeaker 120A and the second loudspeaker 120B are operated as speakers capable of providing stereo sounds to the user wearing the head-mounted device 110.
The control device 130 is configured to provide video signal to the head-mounted device 110 to drive the display module 112, and to modify a first audio signal asA and a second audio signal asB (depicted in
The first loudspeaker 120A and the second loudspeaker 120B are similar to each other, and therefore only the components and connection relationships of the first loudspeaker 120A are described in detail below. The first loudspeaker 120A comprises a communication interface 230, a position tracking circuit 240 and an audio output circuit 250. The communication interface 230 is configured to communicate with the control device 130 to receive the filtered first audio signal F_asA therefrom. In some embodiments, the communication interface 230 is configured to communicate with the communication interface 210 of the head-mounted device 110 to indirectly receive the filtered first audio signal F_asA via the head-mounted device 110. The position tracking circuit 240 is configured to generate position information and orientation information to be processed by the control device 130 so that the control device 130 may determine the position and orientation of the first loudspeaker 120A relative to the head-mounted device 110. The audio output circuit 250 is configured to generate sounds according to the filtered first audio signal F_asA.
In some embodiments, the communication interfaces 210 and 230 may be wired or wireless interfaces, such as Bluetooth, ZigBee or Ethernet.
In some embodiments, the position tracking circuits 220 and 240 may comprise a plurality of optical sensors configured to sense invisible light (e.g., the infrared light) emitted by a plurality of base stations (e.g., the lighthouses) arranged in the physical environment.
In other embodiments, the position tracking circuits 220 and 240 may be radio-frequency (RF) transceivers suitable for ultra-wideband positioning. For example, the position tracking circuits 220 and 240 may communicate with each other by ultra-wideband signals, so that the position and orientation of the first loudspeaker 120A relative to the head-mounted device 110 can be obtained by the time-of-flight method.
The control device 130 is configured to receive the first audio signal asA and the second audio signal asB, in which the first audio signal asA and the second audio signal asB carry audio data of the first loudspeaker 120A and the second loudspeaker 120B, respectively. The control device 130 is further configured to apply one or more filters to the first audio signal asA and the second audio signal asB according to the connection status of the first loudspeaker 120A and the second loudspeaker 120B (i.e., coupled to or detached from the head-mounted device 110), in order to alter the first audio signal asA and the second audio signal asB at one or more frequencies. Such filters include, but are not limited to, a headphone effect filter 23, a loudspeaker effect filter 24, a position compensation filter 25, a crosstalk cancellation filter 26 and a head-related transfer function (HRTF) filter 27, which may be stored in a memory that can be accessed by the control device 130.
In operation S301, position information and orientation information of the head-mounted device 110, the first loudspeaker 120A and the second loudspeaker 120B are obtained, for example, through the position tracking circuits 220 and 240. In some embodiments, one or more sensors, such as accelerometers and gyroscopes, may be incorporated in these devices of the system 100 in assistance to provide the orientation information.
In operation S302, it is determined that whether the first loudspeaker 120A and the second loudspeaker 120B are physically coupled to the head-mounted device 110. For example, the control device 130 may receive and process the position information and the orientation information to determine the positions of the first loudspeaker 120A and the second loudspeaker 120B relative to the head-mounted device 110. The control device 130 may select the filters to be applied to the first audio signal asA and the second audio signal asB according to the connection status of the first loudspeaker 120A and the second loudspeaker 120B.
If the first loudspeaker 120A and the second loudspeaker 120B are coupled to the head-mounted device 110 to form a headphone, operations S303-S306 may be conducted to apply at least one of the headphone effect filter 23 and the position compensation filter 25 to the first audio signal asA and the second audio signal asB. On the other hand, if the first loudspeaker 120A and the second loudspeaker 120B are detached from the head-mounted device 110 to be operated as speakers, operations S307-S310 may be conducted to apply at least one of the loudspeaker effect filter 24, the crosstalk cancellation filter 26 and the HRTF filter 27.
In operation S303, the headphone effect filter 23 is applied to the first audio signal asA and the second audio signal asB. The headphone effect filter 23 is configured to mitigate distortion of sounds generated by the first loudspeaker 120A and the second loudspeaker 120B coupled with the head-mounted device 110 (hereinafter referred to as the “headphone configuration”), in which the distortion is at least partially caused by the circuitry of the headphone configuration (i.e., a circuitry comprising the head-mounted device 110, the first loudspeaker 120A and the second loudspeaker 120B coupled with each other).
The first and second audio signals asA and asB filtered by the headphone effect filter 23 may be provided to the first and second loudspeakers 120A and 120B, respectively, as the filtered first and second audio signals F_asA and F_asB in some embodiments, or the first and second audio signals asA and asB may be further processed by one or more of operations S304-S306. By comparing the practical frequency response 420 with the ideal frequency response 440, it is appreciated that sounds generated based on the first and second audio signals asA and asB filtered by the headphone effect filter 23 have mitigated distortions at the entrances of the ear canals of the user compared to sounds generated based on unfiltered audio signals. In specific, the sounds generated based on the first and second audio signals asA and asB filtered by the headphone effect filter 23 have an enhanced (i.e., flattened) frequency response compared to the sounds generated based on the unfiltered audio signals.
In operation S304, whether the first loudspeaker 120A and the second loudspeaker 120B are coupled to correct terminals of the head-mounted device 110 is determined according to the position information and the orientation information. The control device 130 may check whether the positions of the first loudspeaker 120A and the second loudspeaker 120B correspond to the sound channels of the filtered first audio signal F_asA and the filtered second audio signal F_asA.
For example, the filtered first audio signal F_asA may correspond to a right channel, the control device 130 may check whether the first loudspeaker 120A is coupled to the second terminal 116 (e.g., the right terminal corresponding to the right channel. The filtered second audio signal F_asB may correspond to a left channel, the control device 130 may check whether the second loudspeaker 120B is coupled to the first terminal 114 (e.g., the left terminal corresponding to the left channel). If the determination result of operation S304 is “YES,” operation 305 is omitted and operation S306 may be conducted. If the determination result of operation S304 is “NO” (e.g., the headphone configuration of
In operation S305, the filtered first audio signal F_asA and the filtered second audio signal F_asB received by the first loudspeaker 120A and the second loudspeaker 120B, respectively, may be swapped with each other. The control device 130 may, for example, transmit the filtered first audio signal F_asA previously transmitted to the first loudspeaker 120A to the second loudspeaker 120B, and transmit the filtered second audio signal F_asB previously transmitted to the second loudspeaker 120B to the first loudspeaker 120A. Accordingly, the system 100 allows the user to couple the first and second loudspeakers 120A and 120B to the head-mounted device 110 in an arbitrary manner without distorting the sound effect, realizing quick assembling of the headphone configuration to keep the immersive experience.
In operation S306, position compensation may be applied on the first audio signal asA and the second audio signal asB which have been filtered by the headphone effect filter 23.
The ideal frequency response 630 can be seen as a frequency response obtained at an ideal position 640 corresponding to the entrance of the ear canal of the user, and the difference between the practical frequency response 620a and the ideal frequency response 630 is because of a position 650a of the first loudspeaker 120A deviated from the ideal position 640. As shown in
The first and second audio signals asA and asB processed by operations S303-S306 are outputted by the control device 130 as the filtered first and second audio signals F_asA and F_asB, respectively. Accordingly, the user does not require to adjust the first and second loudspeakers 120A and 120B to absolutely correct positions in each time he/she couple the first and second loudspeakers 120A and 120B back to the head-mounted device 110, since the system 100 may automatically compensate the audio according to the user's wearing situation.
Reference is made to
In operation S307, the loudspeaker effect filter 24 is applied to the first audio signal asA and the second audio signal asB. The loudspeaker effect filter 24 is configured to cancel distortions at least partially caused by a circuitry of the speaker configuration (e.g., a circuitry comprising the detached head-mounted device 110, the first loudspeaker 120A and the second loudspeaker 120B) to obtain flatten frequency responses. The coefficients for the first loudspeaker 120A in the loudspeaker effect filter 24 may be generated by an exemplary method including steps of (1) placing the first loudspeaker 120A in a unechoic chamber, (2) obtaining a practical frequency response of sounds generated by the first loudspeaker 120A, and (3) obtain filter coefficients for the first loudspeaker 120A by an adaptive filter similar to the one discussed with reference to
Different distances between the user and the first loudspeaker 120A may cause different frequency responses, and may require different level of filtering. In some embodiments, multiple of sets of coefficients of the loudspeaker effect filter 24 may be generated by the above method, and the control device 130 may select a set of coefficients as the coefficients for the first loudspeaker 120A in the loudspeaker effect filter 24 according to a distance between the first loudspeaker 120A and the head-mounted device 110. Coefficients for the second loudspeaker 120B in the loudspeaker effect filter 24 may be generated in a similar fashion, and therefore those descriptions are omitted.
The first and second audio signals asA and asB filtered by the loudspeaker effect filter 24 may be provided to the first and second loudspeakers 120A and 120B, respectively, as the filtered first and second audio signals F_asA and F_asB in some embodiments, or the first and second audio signals asA and asB may be further processed by one or more of operations S308-S310.
In operation S308, it is determined that whether the first loudspeaker 120A and the second loudspeaker 120B are in positions corresponding to the sound channels of the filtered first audio signal F_asA and the filtered second audio signal F_asB they received.
In operation S309, the filtered first audio signal F_asA and the filtered second audio signal F_asB received by the first loudspeaker 120A and the second loudspeaker 120B, respectively, may be swapped with each other.
In operation S310, the crosstalk cancellation filter 26 and the HRTF filter 27 are applied to the first audio signal asA and the second audio signal asB filtered by the loudspeaker effect filter 24. The crosstalk cancellation filter 26 may render the first loudspeaker 120A and the second loudspeaker 120B act like they are in the headphone configuration to provide life-like binaural sounds. In the situation of
Positions and orientations of a speaker relative to the user may influence the interaural time difference (ITD), the interaural level difference (ILD) and the frequency response. Therefore, in some embodiments, the control device 130 may obtain coefficients of the crosstalk cancellation filter 26 and the HRTF filter 27 according to the positions and orientations of the head-mounted device 110, the first loudspeaker 120A and the second loudspeaker 120B, by an adaptive filter similar to the one discussed with reference to
The first and second audio signals asA and asB processed by operations S307-S310 may be outputted by the control device 130 as the filtered first and second audio signals F_asA and F_asB, respectively. Accordingly, the system 100 allows the user to place the first loudspeaker 120A and the second loudspeaker 120B in arbitrary positions and orientations without distorting the sound effect, realizing quick disposing of the speaker configuration to keep the immersive experience. In addition, the speaker configuration allows sounds of the physical environment to be heard by the user, and can broadcast sounds to other people, which helps to improve communication efficiency in various scenarios (e.g., meeting or gaming).
Certain terms are used throughout the description and the claims to refer to particular components. One skilled in the art appreciates that a component may be referred to as different names. This disclosure does not intend to distinguish between components that differ in name but not in function. In the description and in the claims, the term “comprise” is used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to.” The term “couple” is intended to compass any indirect or direct connection. Accordingly, if this disclosure mentioned that a first device is coupled with a second device, it means that the first device may be directly or indirectly connected to the second device through electrical connections, wireless communications, optical communications, or other signal connections with/without other intermediate devices or connection means.
The term “and/or” may comprise any and all combinations of one or more of the associated listed items. In addition, the singular forms “a,” “an,” and “the” herein are intended to comprise the plural forms as well, unless the context clearly indicates otherwise.
Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the present disclosure being indicated by the following claims.
Number | Name | Date | Kind |
---|---|---|---|
5917916 | Sibbald | Jun 1999 | A |
9277343 | Alexandrov | Mar 2016 | B1 |
11451922 | Moses | Sep 2022 | B1 |
20140334657 | Guenther | Nov 2014 | A1 |
20160366502 | Morris | Dec 2016 | A1 |
20170078821 | Faller | Mar 2017 | A1 |
20180020312 | Visser | Jan 2018 | A1 |
20190387299 | Evans | Dec 2019 | A1 |
20200021940 | Choueiri | Jan 2020 | A1 |
20200103513 | Knaappila | Apr 2020 | A1 |
20200245047 | Ou | Jul 2020 | A1 |
20200372766 | Biggs | Nov 2020 | A1 |
Number | Date | Country |
---|---|---|
106507253 | Mar 2017 | CN |
111474710 | Jul 2020 | CN |
I687106 | Mar 2020 | TW |
202127916 | Jul 2021 | TW |
202147300 | Dec 2021 | TW |
Entry |
---|
Corresponding Taiwan office action dated Feb. 22, 2023. |
Number | Date | Country | |
---|---|---|---|
20230171542 A1 | Jun 2023 | US |