The present invention relates to a communication device, a communication system, a communication control method, and a program.
For communication systems, such as a TV conference system, that realize a remote conference using a communication network, there is a technology for transmitting/receiving encoded data obtained through encoding using a scalable coding format between communication devices serving as terminals. For example, Patent Document 1 discloses a TV conference system including a transmitting terminal that transmits encoded data obtained through scalable coding using the H.264/SVC format, a receiving terminal that receives and decodes the encoded data, and a conference bridge connecting the transmitting terminal and the receiving terminal. In the TV conference system described in Patent Document 1, when relaying the encoded data from the transmitting terminal and transmitting the encoded data to the receiving terminal, the conference bridge determines the network state of the receiving terminal. The conference bridge then chooses encoded data of a video stream in quality suitable to the state of the receiving terminal from among the encoded data that is obtained through scalable coding in the transmitting terminal and then transmitted and transmits the chosen encoded data to the receiving terminal.
In the conventional technology, however, the authority to determine the state of the receiving terminal to choose, with respect to the encoded data obtained through scalable coding in the transmitting terminal, encoded data to be actually transmitted to the receiving terminal exists in a relay device, such as the conference bridge described in Patent Document 1. For this reason, regardless of the state of the receiving terminal, the transmitting terminal transmits all the encoded data obtained through scalable coding according to a given setting to the relay device and data in quality normally unnecessary may be transmitted to the relay device. In this case, the network bandwidth of the transmitting terminal is used more than necessary.
To solve the above-described problem, one aspect of the present invention is a communication device configured to transmit/receive encoded data that is obtained through scalable coding to/from at least one other communication device, decode the received encoded data, and reproduce and output the data, the communication device including a notification unit configured to notify the at least one other communication device from which data being reproduced and output is transmitted, of reproduction quality information representing quality of the data being reproduced and output; and an encoding setting control unit configured to control a setting for encoding the encoded data to be transmitted to the at least one other communication device from which reproduction quality information is notified, based on the reproduction quality information that is notified from the at least one other communication device.
The present invention provides an effect that a setting for scalable coding performed by the transmitting terminal is controlled according to the state of the receiving terminal, and the use of the network bandwidth of the transmitting terminal more than necessary can be prevented.
With reference to the accompanying drawings, an embodiment of the communication device, the communication system, the communication control method, and the program according to the present invention will be described in detail below. As an exemplary communication system to which the invention is applied, a TV conference system (also referred to as a “video conference system”) will be exemplified, which is a TV conference system that transmits/receives video image data and audio data between multiple TV conference terminals (corresponding to “communication devices”) to realize a remote conference at multiple bases. In the TV conference system, scalable coding is performed on video image data that is captured by a TV conference terminal, the encoded data is transmitted to another TV conference terminal, and the other TV conference terminal decodes the encoded data and reproduces and outputs the data. Note that communication systems to which the invention can be applied are not limited to this example. The present invention can be applied widely to various communication systems that transmit/receive encoded data obtained through scalable coding between multiple communication devices and to various communication terminals that are used in the communication systems.
As illustrated in
The display 11 has a wired or wireless connection to the terminal 10. The display 11 may be configured to be integrated with the terminal 10.
The terminals 10 and the relay servers 30 are connected to routers as, for example, nodes of local area networks (LAN). The router is a network device that chooses a route for data transmission. In the example illustrated in
It is assumed that the LAN 2a and the LAN 2b are constructed at different locations in an area X and the LAN 2c and the LAN 2d are constructed at different locations in an area Y. For example, the area X is Japan, the area Y is the U.S.A., the LAN 2a is constructed in an office in Tokyo, the LAN 2b is constructed in an office in Osaka, the LAN 2c is constructed in an office in N.Y., and the LAN 2d is constructed in an office in Washington D.C. According to the present embodiment, the LAN 2a, the LAN 2b, the dedicated line 2e, the Internet 2i, the dedicated line 2f, the LAN 2c, and the LAN 2d constitute a communication network 2. The communication network 2 may include sections where not only wired communications but also wireless communications according to WiFi (Wireless Fidelity) or Bluetooth (trademark) are performed.
In the TV conference system 1 according to the present embodiment, video image data and audio data are transmitted and received among the multiple terminals 10 via the relay server 30. As illustrated in
The H. 264/SVC (H.264/AVC Annex G) encoding format is known as a standard encoding format for scalable coding on video image data. Through the H. 264/SVC encoding format, video image data is converted into data having a layered structure, the layered-structure data is encoded as a set of multiple sets of video image data that differ in quality, and the sets of encoded data corresponding to the respective sets of video image data can be transmitted and received in multiple channels. According to the present embodiment, the encoded data obtained by encoding the video image data using the 264/SVC encoding format is transmitted and received between the multiple terminals 10.
Specifically, as illustrated in
The quality of video image data includes the resolution that is spatial scalability and the frame rate that is time scalability. The resolution is a screen resolution (also referred to as a screen mode) of a video image that is represented by the number of pixels in the vertical and horizontal directions. According to the present embodiment, both the resolution and the frame rate are dealt with as the quality of video image data. At least one of the resolution and the frame rate of the intermediate-quality video image data is higher than of the low-quality video image data. At least one of the resolution and the frame rate of the high-quality video image data is higher than of the intermediate-quality video image data. According to the present embodiment, the quality of the video image data is classified into low, middle and high three stages. Alternatively, the quality may be classified into two stages or may be finely classified into four or more stages. Furthermore, according to the present embodiment, the quality of video image data containing the resolution and the frame rate is dealt with. Alternatively, the resolution and the frame rate may be dealt with independently. Furthermore, only any one of the resolution and the frame rate may be dealt with as the quality of video image data or another parameter (for example, the S/N ratio), other than the resolution and the frame rate, relating to the quality may be added.
The relay server 30 is a computer that relays transmission of video image data and audio data between the multiple terminals 10. The video image data relayed by the relay server 30 is, as described above, the encoded data obtained through scalable coding using the above-described H.264/SVC format. If the conventional technology is applied to the relay server 30, the relay server 30 would receive sets of encoded data in all the qualities obtained through scalable coding in multiple channels from terminal 10 that transmits the video image data and, in accordance with the network state of the receiving terminal 10 and the display resolution of the video image, choose a channel corresponding to required quality and transmit only the encoded data in that channel to the receiving terminal 10. In the conventional technology, however, because the transmitting terminal 10 transmits the sets of encoded data in all the qualities obtained through scalable coding regardless the state of the receiving terminal 10, the network bandwidth between the transmitting terminal 10 and the relay server 30 is used more than necessary.
For this reason, according to the present embodiment, the transmitting terminal 10 is capable of knowing the quality of a video image that is actually reproduced and output by the receiving terminal 10 (i.e., that is displayed on the display 11) and the transmitting terminal 10 controls the setting for encoding the encoded data to be transmitted to the receiving terminal 10 according to the quality of the video image that is actually reproduced and output by the receiving terminal 10. Thereby, it becomes possible to transmit only the encoded data in the quality required by the receiving terminal 10 from the transmitting terminal 10 via the relay server 30 to the receiving terminal 10 and thus the inconvenience that the network bandwidth between the transmitting terminal 10 and the relay server 30 is used more than necessary can be effectively prevented.
In the conventional technology, as illustrated in
On the other hand, according to the present embodiment, as illustrated in
The notification of the reproduction quality information from the terminal 10 at the base B to the terminal 10 at the base A can be performed using the above-described management information session Sei. The notification of the reproduction quality information from the terminal 10 at the base B to the terminal 10 at the base A may be made via the relay server 30 (via a session different from the data session Sed).
The above-described characteristic processing according to the present embodiment can be implemented by, for example, adding a new function, such as a function of notifying another terminal of the above-described reproduction quality information or a function of controlling the encoding setting on the basis of the reproduction quality information that is notified from another terminal 10, to the terminal 10. The specific configuration example of the terminal 10 including those functions will be described in detail below.
The management server 40 is a computer that manages the overall TV conference system 1 according to the present embodiment. For example, the management server 40 performs management on the state of each terminal 10 that is registered, management of the state of the relay device 30, management of login of the users who use terminals 10, management of the data session Sed that is established between the multiple terminal 10, etc.
Furthermore, according to the present embodiment, as described above, depending on in which quality a video image that is reproduced and output by the receiving terminal 10 is, the transmitting terminal 10 controls the setting for encoding the video image data so as not to transmit video image data in quality higher than required. For this reason, the receiving terminal 10 is not able to reproduce and output video image data in quality exceeding quality of the video image data that is encoded by the transmitting terminal 10; however, when the network state of the receiving terminal 10 improves or when the display layout changes, it may be preferable to reproduce and output video image data in quality higher than quality of the video image data that is currently being output and reproduced. In that case, it is preferable that the management server 40 accept a request for improving the quality of the video image data from the receiving terminal 10 using the above-described management information session Sei and notify the transmitting terminal 10 of the request, and, according to the request from the receiving terminal 10, the transmitting terminal 10 change the encoding setting so as to encode video image data having higher quality.
The program provision server 50 is a computer that provides various programs to the terminals 10, the relay servers 30, the management server 40, and the maintenance server 60. For example, the program provision server 50 stores a terminal program for implementing various functions in the terminals 10 and is capable of transmitting the terminal program to the terminals 10. The program provision server 50 stores a relay server program for implementing various functions in the relay servers 30 and is capable of transmitting the relay server program to the relay servers 30. The program provision server 50 further stores a management server program for implementing various functions in the management server 40 and is capable of transmitting the management server program to the management server 40. The program provision server 50 further stores a maintenance server program for implementing various functions in the maintenance server 60 and is capable of transmitting the maintenance server program to the maintenance server 60.
The maintenance server 60 is a computer that performs maintenance, management and protection of at least one of the terminals 10, the relay servers 30, the management server 40, and the program provision server 50. For example, when the maintenance server 60 is set domestically and the terminals 10, the relay servers 30, the management server 40, or the program provision server 60 is set abroad, the maintenance server 60 remotely performs maintenance, such as maintenance, management and protection of at least one of the terminals 10, the relay servers 30, the management server 40, and the program provision server 50 via the communication network 2.
The hardware configuration of the terminal 10, the relay server 30, the management server 40, and the program provision server 50 of the TV conference system 1 according to the present embodiment will be described.
As illustrated in
The terminal 10 further includes a built-in camera 112 that captures images of a subject to acquire image data according to the control of the CPU 101; an imaging device I/F 113 that controls driving of the camera 112; a built-in microphone 114 that inputs sound; a built-in speaker 115 that outputs sound; an audio input/output I/F 116 that processes the input and output of an audio signal between the microphone 114 and the speaker 115 according to the control of the CPU 101; a display I/F 117 that sends data of a display image to the display 11 according to the control of the CPU 101; an external device connection I/F 118 for connecting various external devices; an alarm lamp 119 that alerts abnormality with respect to the various functions of the terminal 10; and a bus line 110, such as an address bus and a data bus, for electrically connecting the above-described components.
The camera 112, the microphone 114 and the speaker 115 are not necessarily built in the terminal 10, and may be configured to be external components. The display 11 may be configured to be built in the terminal 10. It is assumed that the display 11 is a display device, such as s liquid crystal panel. Alternatively, the display 11 may be a projection device, such as a projector. The hardware configuration of the terminal 10 illustrated in
The terminal program that is provided from the program provision server 50 is stored in, for example, the flash memory 104 and is loaded into the RAM 103 and executed according to the control of the CPU 101. It suffices if the memory for storing the terminal program be a non-volatile memory, and a EEPROM (Electrically Erasable and Programmable ROM), or the like, may be used instead of the flash memory 104. The terminal program may be provided by recording the terminal program in a file in an installable form or an executable form in a recording medium, such as the computer-readable recording medium 106. The terminal program may be provided as a built-in program that is previously stored in, for example, the ROM 102.
As illustrated in
The rely server program that is provided from the above-described program provision server 50 is stored in, for example, the HD 204 and the rely server program is loaded into the RAM 203 and executed according to the control of the CPU 201. The rely server program may be provided by recording the rely server program in a file in an installable form or an executable form in a computer-readable recording medium, such as the recording medium 206 or the CD-ROM 213. The relay server program may be provided as a built-in program that is previously stored in, for example, the ROM 102.
The management server 40 may employ the same hardware configuration as the relay server 30 illustrated in
The program provision server 50 can employ the same hardware configuration as the relay server 30 illustrated in
The maintenance server 60 can employ the same hardware configuration as the relay server 30 illustrated in
As another exemplary detachable recording medium, there is a computer-readable recording medium, such as a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), and a Blu-ray disc. The above-described various programs may be recorded in the recording medium and provided.
The functional configuration of the terminal 10 will be described.
The transmitting and receiving unit 12 transmits/receives various types of data (or information) to/from another terminal 10, the relay server 30, the management server 40, etc., via the communication network 2. The transmitting and receiving unit 12 is implemented with, for example, the network I/F 111 and the CPU 101 illustrated in
The operation input accepting unit 13 accepts various input operations performed by a user who uses the terminal 10. The operation input accepting unit 13 is implemented with, for example, the operation button 108, the power switch 109, and the CPU 101.
The imaging unit 14 captures a video image of the base at which the terminal 10 is set and outputs the video image data. The imaging unit 14 is implemented with, for example, the camera 112, the imaging device I/F 113, and the CPU 101.
The audio input unit 15 inputs sound at the base at which the terminal 10 is set and outputs audio data. The audio input unit 15 is implemented with, for example, the microphone 114, the audio input/output I/F 116, and the CPU 101 that are illustrated in
The audio output unit 16 reproduces and output audio data. The audio output unit 16 is implemented with, for example, the speaker 115, the audio input/output I/F 116, and the CPU 101 that are illustrated in
The encoding unit 17 encodes the video image data that is output from the imaging unit 14 and the audio data that is output from the audio input unit 15 to generate encoded data. Particularly with respect to encoding of video image data, the encoding unit 17 performs scalable coding on video image data according to the H.264/SVC encoding format. The encoding unit 17 is configured to be capable of changing the setting for scalable coding on video image data (for example, setting for the layer structure of data to be encoded) in accordance with a setting signal from the quality control module 25, which will be described below. The encoding unit 17 is implemented, for example, by the CPU 101 illustrated in
The decoding unit 18 decodes the encoded data that is transmitted from another terminal 10 via the relay server 30 and outputs the video image data or the audio data before encoded. The decoding unit 18 is implemented, for example, by the CPU 101 executing the encoding/decoding program (video-image/audio codec) that is contained in the above-described terminal program.
The display video image generation unit 19 uses the video image data decoded by the decoding unit 18 to generate a display video image to be displayed (reproduced and output) on the display 11. For example, when the video image data decoded by the decoding unit 18 contains sets of video image data that are transmitted from multiple terminals 10 at multiple bases, the display video image generation unit 19 generates a display video image containing each of the sets of video image data in a screen according to a predetermined layout setting or a layout setting specified by the user. The display video image generation unit 19 also has a function of passing, to the quality control module 25, which will be described below, information on the layout of the generated display video image, specifically, layout information representing from which base a video image comes and in which size and at which frame rate the video image is contained in the display video image. The display video image generation unit 19 is implemented, for example, by the CPU 101 illustrated in
The display control unit 20 performs control for displaying (reproducing and outputting) the display video image, which is generated by the display video image generation unit 19, on the display 11. The display control unit 20 is implemented with, for example, the display I/F 117 and the CPU 101 illustrated in
The store/read processing unit 21 performs a process of storing/reading various types of data in/from the volatile storage unit 22 or the non-volatile storage unit 23. The store/read processing unit 21 is implemented with, for example, the SSD 105 and the CPU 101 that are illustrated in
The quality control module 25 is a module that performs characteristic processing on the terminal 10 according to the present embodiment. The quality control module 25 is implemented, for example, by the CPU 101 illustrated in
On the basis of the above-described layout information that is passed from the display video image generation unit 19, the notification unit 26 generates the reproduction quality information representing the quality of a video image from another base that is displayed by the display 11 as a display video image. The notification unit 26 then notifies, via the transmitting and receiving unit 12, the other terminal 10 from which the video image is transmitted of the generated reproduction quality information.
In the example in
In the example in
In the example in
The notification unit 26 of the terminal 10 may generate the reproduction quality information described above in a general-purpose form, such as the XML (Extensible Markup Language) form, or in a unique form that is interpretable by the terminals 10.
In the example illustrated in
Furthermore, in the example in
Furthermore, in the example illustrated in
This allows the terminal 10 at each base to know in which quality the video image transmitted to the terminal at another base is reproduced and output at the other base. The notification of the reproduction quality information may be made using, for example, the management information session Sei illustrated in
The encoding setting control unit 27 controls the setting for encoding performed by the encoding unit 17 on the basis of the reproduction quality information that is notified from the terminal 10 at the other base. For example, when the quality represented by the reproduction quality information notified from the terminal 10 at the other base (i.e., the quality of the video image from the own base that is displayed at the other base) is lower than the quality of the encoded data that is transmitted to the terminal 10 at the other base via the relay server 30 (i.e., the maximum quality of encoded data that is generated according to the current setting for encoding performed by the encoding unit 17), the encoding setting control unit 27 changes the setting for encoding performed by the encoding unit 17 (for example, the setting for layer structure) such that quality exceeding the quality represented by the reproduction quality information notified from the terminal at the other base is not contained. In a case where there are multiple other bases to which the video image is being transmitted and common encoded data is transmitted to the terminals 10 at the respective multiple bases, when the highest quality among qualities represented by multiple sets of reproduction quality information that are notified from the terminals 10 at the respective bases is lower than the quality of the common encoded data that is transmitted to the terminals 10 at the respective bases via the relay server 30, the encoding setting control unit 27 changes the setting for encoding performed by the encoding unit 17 such that the quality exceeding the highest quality among the qualities represented by the multiple sets of reproduction quality information is not contained.
In the example in
Then the reproduction quality information containing the items of Resolution: 640×360 and Frame Rate: 30 fps is notified from the terminal 10 at the base B to the terminal 10 at the base A and the reproduction quality information containing the items of Resolution: 320×180 and Frame Rate: 15 fps is notified from the terminal 10 at the base C to the terminal 10 at the base A. The encoding setting control unit 27 of the terminal 10 at the base A acquires the reproduction quality information notified from the terminal 10 at the base B and the reproduction quality information notified from the terminal 10 at the base C and, from these sets of reproduction quality information, determines that there is not any base at which a high-quality video image in a resolution of 1280×720 and at a frame rate of 30 fps is displayed. The encoding setting control unit 27 of the terminal 10 at the base A changes, for example, the setting for the layer structure in encoding performed by the encoding unit 17 such that the layer corresponding to the high-quality video image in a resolution of 1280×720 and at a frame rate of 30 fps is not contained.
Accordingly, encoded data containing only a low-quality video image and an intermediate-quality video image is thereafter transmitted to the relay server 30 from the terminal 10 at the base A, which makes it possible to effectively prevent inconvenience that the bandwidth of the network between the base A and the relay server 30 is used more than necessary. In the example in
At step S103, the encoding setting control unit 27 then determines whether encoded data of a video image exceeding the maximum resolution detected at step S102 is being transmitted to the relay server 30. For example, the encoding setting control unit 27 makes this determination by checking whether the current setting for the layer structure of the encoding unit 17, which performs scalable coding on the video image data, is a setting in which the layer of a video image exceeding the maximum resolution that is detected at step S102 is contained. When encoded data of a video image exceeding the maximum resolution is being transmitted to the relay server 30 (YES at step S103), the encoding setting control unit 27 moves to step S104 and, when no encoded data of a video image exceeding the maximum resolution is being transmitted to the relay server 30 (NO at step S103), the encoding setting control unit 27 moves to step S106.
At step S104, the encoding setting control unit 27 determines whether it is possible to delete the layer corresponding to the video image exceeding the maximum resolution that is detected at step S102. When it is possible to delete the layer corresponding to the video image exceeding the maximum resolution (YES at step S104), the encoding setting control unit 27 moves to step S105. When it is not possible to delete the layer corresponding to the video image exceeding the maximum resolution (NO at step S104), the encoding setting control unit 27 moves to step S106. The case where it is not possible to delete the layer corresponding to the video image exceeding the maximum resolution is, for example, a case where the layer corresponding to the video image exceeding the maximum resolution is a layer containing the resolution equal to or smaller than the maximum resolution, i.e., the resolution of the video image of the own base that is being displayed at another base.
At step S105, the encoding setting control unit 27 changes the setting for the layer structure of the encoding unit 17 such that the layer exceeding the maximum resolution detected at step S102 is not contained and then moves to step S106.
At step S106, the encoding setting control unit 27 determines whether encoded data of the video image exceeding the maximum frame rate detected at step S102 is being transmitted to the relay server 30. For example, the encoding setting control unit 27 makes this determination by checking whether the current setting for the layer structure of the encoding unit 17 that performs scalable coding on the video image data is a setting in which a layer of the video image exceeding the maximum frame rate detected at step S102 is contained. When encoded data of the video image exceeding the maximum frame rate is being transmitted to the relay server 30 (YES at step S106), the encoding setting control unit 27 moves to step S107. When no encoded data of the video image exceeding the maximum frame rate is being transmitted to the relay server 30 (NO at step S106), the encoding setting control unit 27 ends the process represented in the flowchart of
At step S107, the encoding setting control unit 27 changes the setting for the layer structure of the encoding unit 17 such that the layer exceeding the maximum frame rate that is detected at step S102 is not contained and ends the process represented by the flowchart of
First of all, the encoded data of the low-quality video image, the intermediate-quality video image, and the high-quality video image is transmitted from the transmitting terminal 10 to the receiving terminal 10 via the relay server 30 (step S201). In the receiving terminal 10, the decoding unit 18 decodes the received encoded data and the display video image generation unit 19 generates a display image to be displayed on the display 11 (step S202). The display video image generation unit 19 passes layout information on the generated display video image to the notification unit 26.
The notification unit 26 of the receiving terminal 10 generates reproduction quality information on the basis of the layout information that is passed from the display video image generation unit 19 (step S203) and notifies the transmitting terminal 10 of the generated reproduction quality information (step S204). In this example, the reproduction quality information notified to the transmitting terminal 10 represents that the quality of the video image being displayed by the receiving terminal 10 on the display 11 is in intermediate quality.
In the transmitting terminal 10, when the transmitting terminal 10 is notified of the reproduction quality information by the receiving terminal 10, the encoding setting control unit 27 determines whether it is necessary to change the setting for encoding performed by the encoding unit 17 on the basis of the reproduction quality information (step S205). In this example, because it is not necessary to transmit the high-quality video image to the receiving terminal 10, the encoding setting control unit 27 determines that it is necessary to change the setting for encoding performed by the encoding unit 17. The encoding setting control unit 27 then changes the setting for the layer structure for encoding performed by the encoding unit 17 such that the layer corresponding to the high-quality video image is not contained (step S206).
The encoded data of the low-quality video image and the intermediate-quality video image obtained through encoding performed by the encoding unit 17 whose setting is changed is then transmitted to the receiving terminal 10 from the transmitting terminal 10 via the relay server 30 (step S207). Thereby, the inconvenience that the bandwidth of the network between the transmitting terminal 10 and the relay server 30 is used more than necessary can be effectively prevented.
As descried in detail above by giving the specific example, in the TV conference system 1 according to the present embodiment, the receiving terminal 10 notifies the transmitting terminal 10 from which the video image is transmitted of the reproduction quality information representing the quality of the video image that is actually being reproduced and output. On the basis of the reproduction quality information that is notified to the transmitting terminal 10 from the receiving terminal 10, the transmitting terminal 10 controls the setting for encoding the video image to be transmitted to the receiving terminal 10. Accordingly, the transmitting terminal 10 is able to encode only the video image in quality required by the receiving terminal 10 and transmit the video image to the relay server 30 and effectively prevent the inconvenience that the network bandwidth between the transmitting terminal 10 and the relay server 30 is used more than necessary.
A second embodiment will be described here. In the present embodiment, the notification unit 26 of the terminal 10 generates reproduction quality information containing the compression ratio of a display video image and notifies another terminal 10 from which the video image is transmitted of the reproduction quality information. According to the present embodiment, when generating a display image to be displayed on the display 11 using the video image data that is decoded by the decoding unit 18, the display video image generation unit 19 generates information containing the compression ratio in addition to the size (resolution) and frame rate of the video image from each base as information on the layout of the generated display image and passes the information to the quality control module 25. On the basis of the layout information that is passed from the display video image generation unit 19, the notification unit 26 of the quality control module 25 generates reproduction quality information containing the compression ratio in addition to the resolution and the frame rate as reproduction quality information representing the quality of the video image from another base that is displayed by the display 11 as the display video image and notifies, via the transmitting and receiving unit 12, another transmitting terminal 10 from which the video image is transmitted of the reproduction quality information. The compression ratio in the reproduction quality information represents at which ratio the video image data of the video image from each base that is contained in the display video image is compressed and transmitted, and the compression ratio in a case where the video image that is compressed to be halved in volume is contained in the display video image is 50% and the compression ratio in a case where an uncompressed video image is contained in the display video image is 100%.
In the example in
In the example in
In the example in
In the example illustrated in
Furthermore, in the example in
Furthermore, in the example illustrated in
In the example in
Then the reproduction quality information containing the items of Resolution: 640×360, Frame Rate: 30 fps, and Compression ratio: 100% is notified from the terminal 10 at the base B to the terminal 10 at the base A and the reproduction quality information containing the items of Resolution: 320×180, Frame Rate: 15 fps, and Compression ratio: 50% is notified from the terminal 10 at the base C to the terminal 10 at the base A. The encoding setting control unit 27 of the terminal 10 at the base A acquires the reproduction quality information notified from the terminal 10 at the base B and the reproduction quality information notified from the terminal 10 at the base C and, from these sets of reproduction quality information, determines that there is not any base at which a high-quality video image in a resolution of 1280×720, at a frame rate of 30 fps, and at a compression ratio of 100% is displayed. The encoding setting control unit 27 of the terminal 10 at the base A changes, for example, a setting for the layer structure in encoding performed by the encoding unit 17 such that the layer corresponding to the high-quality video image in a resolution of 1280×720, at a frame rate of 30 fps, and at a compression ratio of 100% is not contained.
Accordingly, encoded data containing only low-quality video images and intermediate-quality video images is transmitted to the relay server 30 from the terminal 10 at the base A thereafter, which makes it possible to effectively prevent the inconvenience that the bandwidth of the network between the base A and the relay server 30 is used more than necessary.
In the example in
In the example in
According to the present embodiment, when the result of the determination at step S306 in
At step S308, the encoding setting control unit 27 determines whether the encoded data of the video image exceeding the maximum compression ratio that is detected at step S302 is being transmitted to the relay server 30. For example, the encoding setting control unit 27 makes this determination by checking whether the current setting for the layer structure of the encoding unit 17 that performs scalable coding on the video image data is a setting in which the layer of the video image exceeding the maximum compression ratio that is detected at step S302 is contained. When encoded data of the video image exceeding the maximum compression ratio is being transmitted to the relay server 30 (YES at step S308), the encoding setting control unit 27 moves to step S309. When no encoded data of the video image exceeding the maximum compression ratio is being transmitted to the relay server 30 (NO at step S308), the process represented by the flowchart of
At step S309, the encoding setting control unit 27 changes the setting for the layer structure of the encoding unit 17 such that the layer corresponding to the video image exceeding the maximum compression ratio that is detected at step S302 is not contained and then ends the process represented in the flowchart in
In the above-described example, the resolutions, the frame rates and the compression ratios of the video image and the input data from the external device that are actually being displayed on the display 11 are dealt with as the reproduction quality information. Alternatively, the resolution, the frame rate, and the compression ratio that are different from the quality of the video image and the data that are contained in the display video image actually displayed on the display 11 but corresponds to the reproduction quality required by the user as the quality of the video image to be displayed on the display video image, for example, may be dealt with as the reproduction quality information. In this case, for example, the notification unit 26 of the terminal 10 accepts an operation of the user that specifies a video image to be displayed on a display video image, a resolution, a frame rate, or a compression ratio of data, generates reproduction quality information corresponding to the quality specified by the user, and notifies the transmitting terminal 10 of the video image and the data.
In this example, the terminal 10 from which the video image and the data is transmitted may be notified of the reproduction quality information that specifies quality exceeding the high-quality video image contained in the encoded data that is being transmitted by the terminal 10. In this case, the encoding setting control unit 27 of the terminal 10 changes the setting for encoding performed by the encoding unit 17 such that the quality specified by the reproduction quality information notified from another terminal 10 is contained in the encoded data to be transmitted from the terminal 10 to the other terminal 10.
Then the reproduction quality information containing the items of Resolution: 1280×720, Frame Rate: 30 fps, and Compression ratio: 100% is notified from the terminal 10 at the base B to the terminal 10 at the base A and the reproduction quality information containing the items of Resolution: 320×180, Frame Rate: 15 fps, and Compression ratio: 50% is notified from the terminal 10 at the base C to the terminal 10 at the base A. The encoding setting control unit 27 of the terminal 10 at the base A acquires the reproduction quality information notified from the terminal 10 at the base B and the reproduction quality information notified from the terminal 10 at the base C and, from these sets of reproduction quality information, determines that there is a request for displaying, at the base B, a high-quality video image in a resolution of 1280×720, a frame rate of 30 fps, and a compression ratio of 100%. In this case, the encoding setting control unit 27 of the terminal 10 at the base A changes, for example, the setting for the layer structure in coding performed by the encoding unit 17 such that the layer corresponding to the high-quality video image in a resolution of 1280×720, at a frame rate of 30 fps, and at a compression ratio of 100% is added.
When, as in the above-described case, the resolution, the frame rate, the compression ratio, etc., corresponding to the reproduction quality required by the user are dealt with as the reproduction quality information, the required resolution, frame rate, and compression ratio may be specified by a range. In this case, the notification unit 26 of the terminal 10 accepts an operation of the user for specifying a range of, for example, resolution, frame rate and compression ratio of a video image and data to be displayed on the display, generates reproduction quality information containing the quality range that is specified by the user, and notifies the terminal 10 from which the video image and the data are transmitted.
At step S403, the encoding setting control unit 27 then determines whether encoded data covering the maximum resolution detected at step S402 is being transmitted to the relay server 30. When the encoded data being transmitted to the relay server 30 does not cover the maximum resolution detected at step S402 (NO at step S403), the encoding setting control unit 27 moves to step S404. When the encoded data covers the maximum resolution (YES at step S403), the encoding setting control unit 27 moves to step S406.
At step S404, the encoding setting control unit 27 determines whether it is possible to add the layer corresponding to the maximum resolution that is detected at step S402. When it is possible to add the layer corresponding to the maximum resolution (YES at step S404), the encoding setting control unit 27 moves to step S405. When it is not possible to add the layer corresponding to the maximum resolution (NO at step S404), the encoding setting control unit 27 moves to step S406.
At step S405, the encoding setting control unit 27 changes the setting for the layer structure of the encoding unit 17 so as to add the layer corresponding to the maximum resolution that is detected at step S402 and then moves to step S406.
At step S406, the encoding setting control unit 27 determines whether the encoded data covering the maximum frame rate that is detected at step S402 is being transmitted to the relay server 30. When the encoded data being transmitted to the relay server 30 does not cover the maximum frame rate detected at step S402 (NO at step S406), the encoding setting control unit 27 moves to step S407. When the encoded data covers the maximum frame rate (YES at step S406), the encoding setting control unit 27 moves to step S409.
At step S407, the encoding setting control unit 27 determines whether it is possible to add the layer corresponding to the maximum frame rate that is detected at step S402. When it is possible to add the layer corresponding to the maximum frame rate (YES at step S407), the encoding setting control unit 27 moves to step S408. When it is not possible to add the layer corresponding to the maximum frame rate (NO at step S407), the encoding setting control unit 27 moves to step S409.
At step S408, the encoding setting control unit 27 changes the setting for the layer structure of the encoding unit 17 so as to add the layer corresponding to the maximum frame rate that is detected at step S402 and then moves to step S409.
At step S409, the encoding setting control unit 27 determines whether encoded data covering the maximum compression ratio that is detected at step S402 is being transmitted to the relay server 30. When the encoded data being transmitted to the relay server 30 does not cover the maximum compression ratio detected at step S402 (NO at step S409), the encoding setting control unit 27 moves to step S410. When the encoded data covers the maximum compression ratio (YES at step S409), the encoding setting control unit 27 ends the process represented by the flowchart of
At step S410, the encoding setting control unit 27 determines whether it is possible to add the layer corresponding to the maximum compression ratio that is detected at step S402. When it is possible to add the layer corresponding to the maximum compression ratio (YES at step S410), the encoding setting control unit 27 moves to step S411. When it is not possible to add the layer corresponding to the maximum compression ratio (NO at step S410), the encoding setting control unit 27 ends the process represented by the flowchart of
At step S411, the encoding setting control unit 27 changes the setting for the layer structure of the encoding unit 17 so as to add the layer corresponding to the maximum compression ratio detected at step S402 and then ends the process represented by the flowchart of
The specific embodiments of the present invention have been described above; however, the present invention is not limited to the above-described embodiments and, when carried out, can be embodied by adding various modifications and changes within the scope of the invention. In other words, the specific configurations and operations of the TV conference system 1, the terminal 10, etc., that are described in the above-described embodiments are examples only and various modifications may be made according to the use and purpose.
For example, according to the above-described embodiments, the terminal 10 includes the quality control module 25 including the notification unit 26 and the encoding setting control unit 27. Alternatively, another device, such as the management server 40, may include part or all the functions of the quality control module 25. For example, with respect to the configuration in which the management server 40 includes the notification unit 26, the management server 40 acquires layout information from the receiving terminal 10, generates reproduction quality information on the basis of the layout information, and notifies the transmitting terminal 10 of the reproduction quality information. Furthermore, with respect to the configuration in which the management server 40 includes the encoding setting control unit 27, the management server 40 acquires reproduction quality information that is notified from the receiving terminal 10, generates a control signal that controls the setting for the encoding unit 17 of the transmitting terminal 10 on the basis of the reproduction quality information, and transmits the control signal to the transmitting terminal 10.
According to the above-described embodiment, scalable coding is performed on video image data and the encoded video image data is transmitted and received between terminals 10. Alternatively, scalable coding may be performed on audio data together with or instead of the video image data and the encoded data may be transmitted and received between terminals 10. In this case, the quality of the audio data may include, for example, the sampling frequency of a sound and the bit rate of the sound.
Furthermore, according to the above-described embodiments, the TV conference system 1 is exemplified as an exemplary communication system to which the present invention is applied, which is however not a limitation. For example, the present invention is effectively applicable to various communication systems that are, for example, a telephone system, such as an IP (Internet Protocol) phone with which audio data is bi-directionally transmitted and received between terminals, and a car navigation system in which geographic data, route information, etc., are distributed from a terminal at a management center to a car navigation device that is mounted on a vehicle.
Furthermore, according to the above-described embodiments, the TV conference terminal (terminal) 10 is exemplified as an exemplary communication device to which the present invention is applied, which is however not a limitation. The present invention is effectively applicable to various communication devices, such as a PC, a tablet terminal, a smartphone, an electronic black board, and a car navigation device that is mounted on a vehicle, as long as the communication device has a function of performing scalable coding on various types of data and transmitting the encoded data and a function of decoding the encoded data and reproducing the data.
Patent Literature 1: Japanese Patent No. 4921488
Number | Date | Country | Kind |
---|---|---|---|
2014-035833 | Feb 2014 | JP | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/JP2015/050969 | 1/15/2015 | WO | 00 |