Patent Application
20060045276
This application claims the benefit of a Japanese Patent Application No.2004-254628 filed Sep. 1, 2004, in the Japanese Patent Office, the disclosure of which is hereby incorporated by reference.
1. Field of the Invention
The present invention generally relates to stereophonic reproducing methods, communication apparatuses and computer-readable storage media, and more particularly to a stereophonic reproducing method for reproducing stereophonic sound based on dynamic image information, a communication apparatus which employs such a stereophonic reproducing method, and a computer-readable storage medium which stores a program for causing a computer to reproduce stereophonic sound.
2. Description of the Related Art
Conventionally, as a method of reproducing stereophonic sound, there is a method which embeds in advance position information of a sound source within a dynamic image into information that is sent transmitted from a transmitting end. For example, the position information of a stereophonic sound source within the dynamic image is represented as a difference in right and left volumes of the stereo sound. In addition, in the case of a general stereophonic sound reproducing mechanism, the position information of the sound source is transmitted from the transmitting end which transmits the information, and a reproducing end moves the position of the sound source based on the position information of the sound source. In other words, the position information of the sound source is always transmitted from the transmitting end to the reproducing end in a form added to audio information. For this reason, in the case where the stereophonic sound information such as the position information of the sound source is not included in the information that is transmitted from the transmitting end, the reproducing end cannot reproduce the stereophonic sound.
A Japanese Laid-Open Patent Application No.8-305829 proposes a sound extrapolation method which gives presence by an icon or the like within a still image that is displayed in a main window and extrapolating sound at each moved position of the icon or the like. More particularly, a database of sound data is created in advance, and corresponding sound is reproduced when a user selects a position on a screen by clicking the position by an input device. For example, in the case of a still image of a picture having a stream in front of a forest, the sound of a wind is reproduced when the user clicks the forest, and the sound of flowing water is reproduced when the user clicks the stream.
A Japanese Laid-Open Patent Application No.9-247564 proposes a television receiver that realizes a user benefit function for supporting an audience based on audiovisual information output from a television camera. More particularly, an audience distance is measured by an automatic focusing mechanism of the television camera, and a signal processing is carried out to make an edge-adding contour emphasis and volume adjustment using one characteristic dependent on the audience distance, so as to make an audience support such as making an optimum image display and sound reproduction dependent on the audience distance.
A Japanese Laid-Open Patent Application NO.2002-41038 proposes a virtual musical instrument playing apparatus that synthesizes an image that is picked up by a video camera to an image of a virtual musical instrument, and enables a user to play the virtual musical instrument by moving while watching the synthesized image. More particularly, an operating position of a player for playing the musical instrument is detected, and an image including the virtual musical instrument and the image of the player are synthesized and displayed, so as to create instrument playing information from position information of fingertips of the player when two-dimensional contours of the virtual musical instrument and the player touch each other.
If the audio information transmitted from the transmitting end is monaural sound information and includes no stereophonic sound information, it is possible to add extrapolation information which enables the user to reproduce stereophonic sound at the receiving end (or reproducing end). But in this case, the load on the user is large if the extrapolation information needs to be added manually by the user. In addition, if the extrapolation information is to be generated automatically by measuring the audience distance using the automatic focusing mechanism of the television camera, for example, the sound reproducing system becomes complex and bulky. Moreover, in either case, the extrapolation information is generated at the receiving end (reproducing end) under closed conditions, and it is impossible to generate the position information of the sound source of the transmitting end, thereby making it impossible to reproduce stereophonic sound at the receiving end (reproducing end) by taking into consideration the position information of the sound source of the transmitting end.
Therefore, the conventional stereophonic sound reproducing methods have problems in that stereophonic sound cannot be reproduced at the receiving end (reproducing end) by taking into consideration the position information of the sound source of the transmitting end, unless the audio information transmitted from the transmitting end includes the position information of the sound source of the transmitting end. In other words, in the case of a video cell phone (portable telephone), for example, when the audio information transmitted from the transmitting end is monaural audio information and includes no stereophonic sound information, it is impossible to reproduce stereophonic sound that takes into consideration the position information of the sound source of the transmitting end, even if the receiving end (reproducing end) is provided with the stereophonic sound reproducing mechanism.
Accordingly, it is a general object of the present invention to provide a novel and useful stereophonic sound reproducing method, communication apparatus and computer-readable storage medium, in which the problems described above are suppressed.
Another and more specific object of the present invention is to provide a stereophonic sound reproducing method, a communication apparatus and a computer-readable storage medium, which enables stereophonic sound to be reproduced at a receiving end (reproducing end) by taking into consideration position information of a sound source of a transmitting end, even if audio information transmitted from the transmitting end includes no stereophonic sound information.
Still another object of the present invention is to provide a stereophonic sound reproducing method for receiving audio information and dynamic image information transmitted from a transmitting end and reproducing sound and dynamic image, comprising a generating step generating position information of a sound source of the transmitting end based on the dynamic image information; and a reproducing step reproducing the audio information based on the position information of the sound source, and reproducing stereophonic sound that takes into consideration the position information of the sound source of the transmitting end. According to the stereophonic sound reproducing method of the present invention, it is possible to reproduce stereophonic sound at a receiving end (reproducing end) by taking into consideration the position information of the sound source of the transmitting end, even if the audio information transmitted from the transmitting end includes no stereophonic sound information. In addition, it is possible to reproduce the stereophonic sound by taking into consideration the position information of the sound source of the transmitting end, so as to realize a video telephone function and the like having presence, as long as the receiving end (or reproducing end) is provided with a hardware and/or software applied with the present invention, without the need to provide special hardware and/or software at the transmitting end.
A further object of the present invention is to provide a communication apparatus comprising a receiving part configured to receive audio information and dynamic image information transmitted from a transmitting end; a position information generating part configured to generate position information of a sound source of the transmitting end based on the dynamic image information; and a sound reproducing part configured to reproduce the audio information based on the position information of the sound source, and reproducing stereophonic sound that takes into consideration the position information of the sound source of the transmitting end. According to the communication apparatus of the present invention, it is possible to reproduce stereophonic sound at a receiving end (reproducing end) by taking into consideration the position information of the sound source of the transmitting end, even if the audio information transmitted from the transmitting end includes no stereophonic sound information. In addition, it is possible to reproduce the stereophonic sound by taking into consideration the position information of the sound source of the transmitting end, so as to realize a video telephone function and the like having presence, as long as the receiving end (or reproducing end) is provided with a hardware and/or software applied with the present invention, without the need to provide special hardware and/or software at the transmitting end.
Another object of the present invention is to provide a computer-readable storage medium which stores a program for causing a computer to receive audio information and dynamic image information transmitted from a transmitting end and to reproduce sound and a dynamic image, the program comprising a generating procedure causing the computer to generate position information of a sound source of the transmitting end based on the dynamic image information; and a reproducing procedure causing the computer to reproduce the audio information based on the position information of the sound source, and to reproduce stereophonic sound that takes into consideration the position information of the sound source of the transmitting end. According to the computer-readable storage medium of the present invention, it is possible to reproduce stereophonic sound at a receiving end (reproducing end) by taking into consideration the position information of the sound source of the transmitting end, even if the audio information transmitted from the transmitting end includes no stereophonic sound information. In addition, it is possible to reproduce the stereophonic sound by taking into consideration the position information of the sound source of the transmitting end, so as to realize a video telephone function and the like having presence, as long as the receiving end (or reproducing end) is provided with a hardware and/or software applied with the present invention, without the need to provide special hardware and/or software at the transmitting end.
Other objects and further features of the present invention will be apparent from the following detailed description when read in conjunction with the accompanying drawings.
A description will be given of embodiments of a stereophonic sound reproducing method, a communication apparatus and a computer-readable storage medium according to the present invention, by referring to the drawings.
A communication apparatus shown in
When the communication apparatus operates as a transmitting end, the modem 3 modulates audio information and dynamic image information that are to be transmitted from the communication apparatus to a receiving end into a format conforming to a communication protocol, and the transmitting and receiving unit 4 transmits modulated information to the receiving end via a wireless (or radio) telephone line (not shown). On the other hand, when the communication apparatus operates as the receiving end, the transmitting and receiving unit 4 receives the modulated information from the transmitting end via the wireless telephone line, and the modem 3 demodulates the modulated information into the original audio information and dynamic image information depending on the communication protocol. A modem having a known structure for realizing the above modulating and demodulating functions may be used for the modem 3. Similarly, a transmitting and receiving unit having a known structure for realizing the above transmitting and receiving functions may be used for the transmitting and receiving unit 4. For the sake of convenience, it is assumed that the communication apparatus on the transmitting end has the functions of the so-called portable telephone having a built-in camera, the audio information is input at the transmitting end by a known method using a microphone or the like, and the dynamic image information is obtained by an image pickup unit or means (not shown), such as the camera, which picks up an image of a target object or the like.
The display unit 5 is formed a liquid crystal display (LCD) or the like, and displays menus and messages for the user when the user operates the communication apparatus, dynamic images of the received dynamic image information, and dynamic images of the dynamic image information that is transmitted. The speaker group 6 includes a plurality of speakers that are arranged to realize the video telephone function and the like having presence, and reproduce stereophonic sound from the received audio information by taking into consideration position information of a sound source. The input device 7 includes keys for inputting numbers and characters, keys for selecting functions, and the like.
In
A step S3 detects, by a known detection method, position information of the target object within the dynamic image that is indicated by the received dynamic image information. The position information of the target object within the dynamic image may be obtained by detecting and tracking, from the contour and the like, the position of the object which occupies an area such that the proportion (or ratio) of the area occupied by the target image with respect to the total area of the dynamic image is greater than or equal to the predetermined value. In addition, the position information of the target object within the dynamic image may be obtained by detecting and tracking a portion that is recognized as a face of a person, such as a portion having skin color.
A step S4 decides whether or not an error is generated at the position of the detected target object 21. In other words, if the target object at the transmitting end is outside an image pickup range that can be picked up by the image pickup unit (or means) and the target object 21 is not visible within the dynamic image 20, the step S4 detects that an error is generated. The process advances to a step S5 if the decision result in the step S4 is NO.
The step S5 generates the position information of the sound source of the transmitting end artificially and continuously, based on a comparison of the registered initial position information of the target object and the position information of the target object detected in the step S3. The position information of the sound source that is generated in the step S5 is obtained from relative coordinates with respect to the initial position information of the object, that is, the center coordinates (0, 0). For this reason, by comparing the position information of the target object that is successively obtained each time with the initial position information, it is possible to generate accurate position information of the sound source by carrying out a relatively simple operation. A step S6 records in the memory 2 the position information of the sound source that is generated in the step S5.
A step S7 supplies to the stereophonic sound mechanism the position information of the sound source recorded in the memory 2, and the process returns to the step S3. The stereophonic sound mechanism subjects the received audio information to a known stereophonic sound process based on the position information of the sound source before supplying the audio information to the speaker group 6. For example, the known stereophonic sound process uses a head-related transfer function (HRTF). Hence, the stereophonic sound is reproduced by taking into consideration the position information of the sound source of the transmitting end. If the decision result in the step S4 is YES, the process advances to the step S7, and thus, the position information of the sound source is not generated in this case, and the stereophonic sound process is carried out based on the position information that is previously recorded in the memory 2.
In this embodiment, the stereophonic sound mechanism is realized by a program stored in the memory 2. Hence, this embodiment of the computer-readable storage medium may store a combination of programs including the program which realizes the stereophonic sound mechanism.
Of course, the stereophonic sound mechanism may be realized by a hardware (semiconductor chip) that carries out a known stereophonic sound process. In this case, the stereophonic sound process can be carried out at a high speed, and a processing load on the CPU 1 can also be reduced. The hardware that carries out the stereophonic sound process may be connected to the bus 8 shown in
For example, the stereophonic sound mechanism may use a software P3D manufactured by SONAPTIC and a semiconductor chip BU7844 manufactured by ROHM, so that a portion of a stereophonic sound algorithm is realized by the semiconductor chip (hardware).
In
When generating the position information of the sound source of the transmitting end, the position information may be generated directly based on the received dynamic image information or, generated based on a dynamic image for display that is obtained by reproducing the received dynamic image information.
In the embodiment described heretofore, the present invention is applied to the portable telephone, and thus, the transmitting end and the receiving end are connected via a wireless telephone line. However, when the present invention is applied to the normal (cable) telephone, the transmitting end and the receiving end are of course connected via the normal telephone line. Moreover, the present invention may be applied to any type of communication apparatus as long as it has a function of communicating the audio information and the image information. Hence, the present invention can also be applied to a personal computer, a data terminal and the like having such a function of communicating the audio information and the image information.
Further, the present invention is not limited to these embodiments, but various variations and modifications may be made without departing from the scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2004-254628 | Sep 2004 | JP | national |
