This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2014-023048, filed on Feb. 10, 2014, the entire contents of which are incorporated herein by reference.
The embodiments discussed herein are related to an information processing device that generates an image for displaying execution results of a computer on a display unit of a terminal device connected via a network and transmits the image to the network, a method, a program, and a terminal device.
In recent years, from the viewpoint of security and Business Continuity Plan (BCP), the use of thin clients has been expanding. Thin client refers to a system architecture in which a client terminal that a user uses is caused to perform minimum processing, and the rest of the processing is concentrated on the side of a server device and the server device is caused to perform the rest of the processing, or refers to a dedicated client terminal device whose functions have been narrowed, which is used in such an architecture.
Accompanying the spread of smart phones and tablets and the speeding up of mobile network, the need for the mobile thin client used by a mobile terminal to securely connect to an in-house system has increased.
In the case where the thin client is used by using a mobile network, there is a problem wherein it is difficult to use the thin client comfortably due to factors such as a change in Network Bandwidth and Round Trip Time (RTT).
Conventionally, in order to solve a problem such as this, a fast screen transfer technique for improving operability by transmitting as a still image a region in which an updating of a screen is not performed frequently and as a moving image a region in which an updating is performed frequently so as to reduce the amount of data when the thin client is used is known (e.g., a technique described in Patent document 1). In this conventional technique, a change frequency determination unit divides an image stored in an image memory into a plurality of regions and determines the frequency of a change between frames for each region. A first image transmission unit transmits an image of a region in which there has been a change. A high-frequency change region identification unit identifies a region whose change frequency has exceeded a threshold value as a high-frequency update region. A transmission stop unit stops the transmission of the identified region by the first image transmission unit. A second image transmission unit transmits the image of the identified region after performing compression processing for a moving image whose compression ratio is higher than that of the first image transmission unit.
Patent document 1: Japanese Laid-open Patent Publication No. 2011-238014
According to an aspect of the embodiments, an information processing device generates an image for displaying execution results of a computer on a display of the terminal device connected via a network and transmits the image to the network. The information processing device includes a processor. The processor is configured to extract a region that is updated as a moving image as a moving image update region from a screen stored in a memory that holds a screen on which an image of execution results of the computer is drawn. The processor is configured to determine a division state of the moving image update region from information including network bandwidth, a threshold value of the transmission time set in advance, an average compression ratio of a frame encoded without using the inter-frame prediction, and an average compression ratio of a frame encoded by using the inter-frame prediction.
The processor is configured to divide the update region that has been determined to be the moving image region in the determined division state. The processor is configured to transmit the divided update region to the terminal device.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention
There has been a problem wherein the time that is needed by a client terminal to complete the reception of all the update region data (moving image data, still image data) corresponding to one frame becomes longer in the case where the network bandwidth is narrow, and therefore, the time from the operation until an image is drawn becomes longer. For example, a simple calculation reveals that, when update region data whose transmission time for one frame via a network having a bandwidth of 100 Mbps (megabit/sec) is 15 milliseconds(msec) is transmitted via a network having a band of 5 Mbps, the transmission time will be 300 msec.
Further, until the reception of all the update region data corresponding to one frame is completed, the screen is not updated, and therefore, the time from the operation of a user until the screen is updated becomes longer and there has been a problem wherein operability deteriorates.
Hereinafter, embodiments for embodying the present embodiments will be explained in detail with reference to the drawings.
A client terminal 120 operates as a thin client terminal for a server 100.
The client terminal 120 includes an operation information acquisition unit 121, a communication unit 122, a screen update information acquisition unit 123, a screen region display unit 124, a high-frequency screen region display unit 125, and a screen display unit 126.
The server 100 includes a communication unit 101, an operation information acquisition unit 102, a display screen generation unit 103, a frame buffer 104, a screen update notification unit 105, a high-frequency screen update region detection unit 106, an update region division unit 107, and a transmission time estimation unit 108. The server 100 further includes a moving image compression ratio estimation unit 109, a division size determination unit 110, an update data generation unit 111, an update region transmission order determination unit 112, and a transfer rate estimation unit 113.
First, in the client terminal 120, the operation information acquisition unit 121 acquires a key input or a mouse operation by a user who operates the client terminal 120 and notifies the communication unit 122 of that as operation information.
Upon receipt of screen update information of the server 100, the communication unit 122 gives the data to the screen update information acquisition unit 123, and sets the reception time to an Ack (Acknowledgement) response indicating an acknowledgement and returns the response to the server 100. Further, upon receipt of the operation information acquired by the operation information acquisition unit 121, the communication unit 122 transmits the operation information to the server 100.
The screen update information acquisition unit 123 acquires update data of the server screen and allocates the update data of the server screen to the high-frequency screen region display unit 125 in the case where the update data is update data of a high-frequency screen region or allocates the update data to the screen region display unit 124 in the case where the update data is update data of regions other than a high-frequency screen region.
The screen region display unit 124 decodes the update data (data of a still-image update region) acquired from the screen update information acquisition unit 123 and writes the update data in the screen data region.
The high-frequency screen region display unit 125 decodes the update data (data of a moving image update region) acquired from the screen update information acquisition unit 123 and writes the update data in the screen data region.
The screen display unit 126 performs drawing of an image on a screen by writing the screen data region in which the update data has been writes in an image drawing memory of a graphics processing unit (GPU).
Next, in the server 100, upon receipt of the operation information in the client terminal 120, the communication unit 101 give the operation information to the operation information acquisition unit 102 and upon receipt of the server screen update data from the transfer rate estimation unit 113, the communication unit 101 transmits the data to the client terminal 120.
The operation information acquisition unit 102 decodes the operation information reported by the communication unit 101 and performs the operation.
The display screen generation unit 103 generates display screen data including an image drawn by an application etc. in response to the operation that was performed by the operation information acquisition unit 102 and writes the display screen data to the frame buffer 104.
The frame buffer 104 performs display processing when the display screen data is written by the display screen generation unit 103.
The screen update notification unit 105 detects an update region in the case where the frame buffer 101 is written and notifies the high-frequency screen update region detection unit 106 of the update region.
When the update region is notified by the screen update notification unit 105, the high-frequency screen update region detection unit 106 sets the update region as a high-frequency screen update region in the case where the number of times of successive updating of the update region is equal to or greater than a threshold value.
The update region division unit 107 notifies the transmission time estimation unit 108 of the region size of the high-frequency screen update region and the region size of the other update regions on the basis of the detection results of the high-frequency screen update region detection unit 106 and acquires the transmission time of each update region from the transmission time estimation unit 108. The update region division unit 107 notifies the division size determination unit 110 of the acquired transmission time of each update region and acquires the division size of the high-frequency screen update region and the division size of the other update regions, respectively, from the division size determination unit 110.
The update region division unit 107 divides the high-frequency screen update region and the other update regions by the acquired division sizes, respectively, and notifies the update region transmission order determination unit 112 of the divided update regions.
Upon receipt of the update region size from the update region division unit 107, the transmission time estimation unit 108 acquires the network band from the transfer rate estimation unit 113, estimates the transmission time, and notifies the update region division unit 107 of the transmission time.
The moving image compression ratio estimation unit 109 acquires the update region size and the compressed moving image size from the update data generation unit 111, and estimates the compression ratio for each of an I-frame and a
P-frame by taking into consideration the past compression ratio and notifies the division size determination unit 110 of the compression ratios.
Here, in the compression of a moving image, the whole of the moving image region is compressed in the first frame (intra-frame compression). In the subsequent frames, compression is performed by also making use of information on the previous frame (inter-frame compression). The size of data that is compressed by the intra-frame compression will be substantially the same as the size of the data that is compressed by still image compression. The frame data compressed by the intra-frame compression is called the I (Intra-coded) frame and the frame data compressed by the inter-frame compression is called the P (Predicted) frame. The compression of moving image data is irreversible compression, and therefore, by periodically performing the intra-frame compression (I-frame) of the moving data, the difference between the compressed data and the original data is inhibited from gradually increasing.
When the transmission time of the update region is reported by the update region division unit 107, the division size determination unit 110 acquires the moving image compression ratio of each of the I-frame and the P-frame from the moving image compression ratio estimation unit 109, determines the division size, and notifies the update region division unit 107 of the division size in the case where the update region is a moving image update region. On the other hand, in the case where the update region is a still-image update region, the division size determination unit 110 determines the division size of the update region so that the transmission time is equal to or less than a threshold value if the reported transmission time exceeds the threshold value.
The update data generation unit 111 encodes each divided update region of the high-frequency screen update region and the other update regions that were divided in accordance with necessity in the update region division unit 107. In the case where the update region is a moving image region, the update data generation unit 111 notifies the moving image compression ratio estimation unit 109 of the region size before the encoding and of the data size after the encoding.
The update region transmission order determination unit 112 determines the transmission order on the basis of the index of the order of priority for each update region and notifies the transfer rate estimation unit 113 of the update region data in the determined order.
The transfer rate estimation unit 113 sets the transmission start time, the transmission data size, etc., for estimating the transfer rate to the header etc. of the transmission data of the update data reported by the update region transmission order determination unit 112 and notifies the communication unit 101 thereof. The transfer rate estimation unit 113 estimates the transfer rate from the reception time that is set when the Ack response is received from the client terminal 120, and the transmission start time and the transmission data size set at the time of transmission.
The above operation of the first embodiment is explained in detail below.
In the present embodiment, the screen update notification unit 105 and the high-frequency screen update region detection unit 106 of the server 100 perform processing by dividing the desktop screen that is stored in the frame buffer 104 into, for example, 8×8 meshes, as illustrated in
The screen update notification unit 105 collects information on squares that are updated on the desktop screen divided into meshes as in
The high-frequency screen update region detection unit 106 estimates screen regions 204 and 205 that have been changed as illustrated in
In the case where the high-frequency screen update region detection unit 106 has determined the update region of the screen to be a still image region other than the high-frequency screen update region, the update region division unit 107 notifies the transmission time estimation unit 108 of the update region size of the still image. Upon receipt of the update region size, the transmission time estimation unit 108 acquires the network band from the transfer rate estimation unit 113, estimates the transmission time, and notifies the update region division unit 107 of the transmission time of the still-image update region. The update region division unit 107 notifies the division size determination unit 110 of the acquired transmission time of the still-image update region. The division size determination unit 110 determines the division size of the still-image update region so that the transmission time is equal to or less than a threshold value in the case where the reported transmission time exceeds the set threshold value and notifies the update region division unit 107 of the division size.
The update data generation unit 111 encodes the divided update region of the still-image update region that has been divided in the update region division unit 107 in accordance with necessity. The still-image update region that has been encoded in this manner is transmitted to the client terminal 120 from the update region transmission order determination unit 112 and the transfer rate estimation unit 113 via the communication unit 101.
In this manner, in the case of the present embodiment, for the update region of a still image, the client terminal 120 sequentially displays the divided update regions in the order of reception, and therefore, the time from the operation until a display is produced is reduced and a user can feel that the response is quick.
Next, the case is explained where the high-frequency screen update region detection unit 106 has determined the update region of the screen to be a moving image update region (high-frequency screen update region).
First, as in the case of the still-image update region, the case is considered where the moving image update region is transmitted by simply dividing the moving image update region on the basis of the transmission time in accordance with necessity.
Because of this, in the present embodiment, in the case where the update region is a moving image, the division size determination unit 110 acquires the moving image compression ratio of each of the I-frame and the P-frame from the moving image compression ratio estimation unit 109 when the transmission time of the moving image update region is reported by the update region division unit 107. Next, the division size determination unit 110 estimates the data amount of the I-frame transmission of the moving image update region from the average compression ratio of the I-frame. Likewise, the division size determination unit 110 estimates the data amount of the P-frame transmission of the moving image update region from the average compression ratio of the P-frame. Then, from the estimated data amounts and transmission time of the I-frame or P-frame transmission, the division size determination unit 110 determines the number of divisions of the update region and the division size so that both the I-frame and the P-frame can be transmitted and notifies the update region division unit 107 of the number of divisions and the division size. The update region division unit 107 divides the I-frame and the P-frame of the moving image update region in the intermingled state by the reported division size and transmits the divided I-frame and P-frame to the client terminal 120 after compressing them as a moving image.
In this manner, in the present embodiment, it is possible to transmit the P-frame of another divided region while the I-frame is being transmitted. Because of this, it is made possible for a user to feel that the display of a moving image in the update region 404 is smooth.
As above, both in the case where the update region is a moving image and in the case where the update region is a still image, in the present embodiment, it is possible for the client terminal 120 to receive the data of the divided region earlier than the moving image data or still image data corresponding to one frame when the network band is narrow. Because of this, it is possible for a user of the client terminal 120 to feel that the time from the operation until an image is drawn is higher than before, and therefore, it is made possible to improve operability.
First, the standby state continues until operation information is received from the client terminal 120 (processing is repeated while the results of the determination at step S501 are NO).
When the operation information is received from the client terminal 120 and the results of the determination at step S501 change to YES, the application program on the server 100, which is specified to be executed by the client terminal 120, stores image information in the frame buffer 104 (step S502). The server 100 performs the operation of the application on the virtual desktop screen that is displayed on the display of the client terminal 120 in such a manner that it seems that a user is operating the desktop screen of the Windows system of the local terminal. The information on this operation is transmitted to the server 100 from client terminal 120. As a result of that, the application program is executed on the server 100 and when rewriting the display of the virtual desktop screen is needed as a result of that, the application program updates the display of the virtual desktop screen region within the frame buffer 104. The processing that is performed at step 5501 corresponds to the functioning of the display screen generation unit 103 in
Next, the screen data of the virtual desktop screen is acquired from the frame buffer 104 (step S503), and whether or not an updating of the screen has occurred is determined (step S504). The processing that is performed at step S503 and step S504 corresponds to the functioning of the screen update notification unit 105 in
In the case where an updating of the screen has not occurred and the results of the determination at step S504 are NO, the processing returns to the standby processing at step S501.
In the case where an updating of the screen has occurred and the results of the determination at step S504 are YES, the network band on which the server 100 is communicating is acquired and the data transfer rate is calculated (step S505).
Next, the amount of data that is transferred is acquired on the basis of the size of the update region of the screen and the compression ratio of the screen data (step S506).
The above-described processing that is performed at step S505 and step S504 corresponds to the functioning of the transmission time estimation unit 108 in
Next, processing to determine whether or not the update region of the screen is a moving image region, or whether or not the update region of the screen is a high-frequency screen update region that should be turned into a moving image, is performed (step S507), and whether or not the update region is a region that has been turned into a moving image (high-frequency screen update region) is determined (step S508).
The processing that is performed at step S507 and step S508 corresponds to the functioning of the high-frequency screen update region detection unit 106.
In the case where the update region is not a region that has been turned into a moving image and the results of the determination at step S508 are NO, the still image region processing described previously by using
After the processing at step S509 or step S510, the processing returns to the standby processing at step S501.
First, from the average compression ratio of the I-frame calculated in the processing at step S606, to be described later, at the transmission timing of the previous moving image update region, the data size when the I-frame is transmitted is estimated (step S601).
Next, from the average compression ratio of the P-frame calculated in the processing at step S607, to be described later, at the transmission timing of the previous moving image update region, the data size when the P-frame is transmitted is estimated (step S601).
Then, from the data amounts of the I-frame transmission or P-frame transmission that have been estimated at step S601 and step S602 are transferred and from the transmission time, the number of divisions of the update region and the division size are determine so that both the I-frame and the P-frame can be transmitted (step S603).
The processing at steps S601 to S603 described above corresponds to the functioning of the division size determination unit 110 in
After that, the data of the divided update region is divided by the division size determined at step S603 and is transmitted to the client terminal 120 after being compressed as a moving image (step S604). This processing corresponds to the functioning of the update region division unit 107 and the update data generation unit 111 in
After that, whether or not the transmitted data is the I-frame is determined (step S605).
In the case where the transmitted data is the I-frame and the results of the determination at step S605 are YES, the average compression ratio of the transmitted I-frame is estimated (step S606). The average compression ratio of the I-frame that is estimated here is referred to in the previously described processing at step S601 at the transmission timing of the next frame of the moving image update region.
In the case where the transmitted data is not the I-frame and the results of the determination at step S605 are NO, the average compression ratio of the transmitted P-frame is estimated (step S607). The average compression ratio of the P-frame estimated here is referred to in the previously described processing at step S602 at the transmission timing of the next frame of the moving image update region.
The processing at step S606 and step S607 described above corresponds to the functioning of the moving image compression ratio estimation unit 109 in
After the processing at step S606 or step S607, the processing of the flowchart in
First, in this processing example, it is assumed that the screen size of the update region represented as 701 in
By the functioning of the division size determination unit 110 in
If the number of divisions is taken to be n, the data amount per transfer is 1/n at the maximum for the I-frame a (n−1)/n at the maximum for the P-frame, and then, the transfer time that is equal to or less than 100 msec, which is the transfer time threshold value, is found. Here, “kbit” means “kilobit”.
((150×8) [kbit]÷n+(15×8) [kbit]×(n−1)÷n)÷5,000 [kbps]≦0.1 [sec]
By the above-described calculation expression, n≧2.84 is found, and therefore, the number of divisions is calculated as three. Consequently, the update region 701 is divided into three regions as illustrated in
In the case where there is a plurality of update regions explained in the first embodiment or in the case where the moving image update region has been changed, setting the order of priority and changing the transmission timing are needed when transmitting the update region from the server to the client terminal. As the case where there is a plurality of update regions, the following cases are assumed as representative cases.
In order to implement control processing in these cases, in the configuration of the server 100 illustrated in
The update region priority determination unit 801 determines the transmission priority of the update region on the basis of the order of priority of the update region detected by the screen update notification unit 105 and the index set in advance, such as the index of the order of priority, and notifies the update region division unit 107 of the transmission priority of the update region.
As parameters that are used as the set indexes, whether or not the window in which the update region is displayed on the virtual desktop screen is an active window, the update region size, whether the update region is a moving image update region or a still-image update region, etc., are used.
In the index example of the order of priority illustrated in
In the case where there is a plurality of update regions having the same order of priority as the results of referring to the index example of the order of priority illustrated in
First, whether or not the update region is within the active window is determined (step S1001).
In the case where the update region is within the active window and the results of the determination at step S1001 are YES, whether or not the update region is a moving image is determined next (step S1002).
In the case where the update region is a moving image and the results of the determination at step S1002 are YES, whether or not the update region is the I-frame is determined further (step S1003).
In the case where the update region is the I-frame and the results of the determination at step S1003 are YES, “1” is set to the order of priority (step S1004). In the case where “1” is set to the order of priority, transmission is performed at each transmission timing as illustrated in
In the case where the update region is the P-frame of a moving image, not the I-frame, and the results of the determination at step S1003 are NO, “2” is set to the order of priority (step S1005). In the case where the update region is within the active window and the results of the determination at step S1001 are YES, and the update region is not a moving image and the results of the determination at step S1002 are NO, i.e., in the case of a still image within the active window, “2” is also set to the order of priority (step S1005). In the case where “2” is set to the order of priority, as in the case where the order of priority is “1”, transmission is performed at each transmission timing as illustrated in
In the case where the update region is not within the active window and the results of the determination at step S1001 are NO, whether or not the update region is a moving image is determined next (step S1006).
In the case where the update region is a moving image and the results of the determination at step S1006 are YES, whether or not the update region is the I-frame is determined further (step S1007).
In the case where the update region is the I-frame and the results of the determination at step S1007 are YES, “3” is set to the order of priority (step S1008). In the case where “3” is set to the order of priority, transmission is performed at a timing at which the I-frame whose order of priority is 1 is not being transmitted, as illustrated in
In the case where the update region is not the I-frame but the P-frame of a moving image and the results of the determination at step S1007 are NO, “4” is set to the order of priority (step S1009). In the case where the update region is not within the active window and the results of the determination at step S1001 are NO, and the update region is not a moving image and the results of the determination at step S1006 are NO, i.e., in the case of a still image within the non-active window, “4” is also set to the order of priority (step S1009). In the case where “4” is set to the order of priority, transmission is performed m times out of n times (m<n) when the I-frame is not being transmitted.
As described above, the functioning of the update region priority determination unit 801 that operates in accordance with the index example of the order of priority in
Next, the transmission timing determination unit 802 in
First, whether or not a moving image update region has been detected is determined (step S1101).
In the case where a moving image update region has been detected and the results of the determination at step S1101 are YES, whether or not a moving image update region whose transmission priority is high has been detected is determined in the update region priority determination processing illustrated in the flowchart in
In the case where a moving image update region whose transmission priority is high has been detected and the results of the determination at step S1102 are YES, the control operation as follows is performed.
First, whether or not the transmission timing of the update region of a new moving image detected at the timing of this time overlaps that of the I-frame of the update region of the already-existing moving image is determined (step S1103).
In the case where the results of the determination at step S1103 are YES, first, the number of divisions of the update region of the new moving image is determined (step S1104). Following this, the number of divisions of the update region of the already-existing moving image is set again (step S1105). After that, the divided update region that is transmitted at the current timing is determined (step S1116). Here, each divided update region is determined so that the data of the divided update region of the new moving image and the data of the divided update region of the already-existing moving image can be transmitted at the same time. After that, the transmission timing determination processing is terminated.
On the other hand, in the case where the transmission timing of the update region of the new moving image does not overlap that of the I-frame of the update region of the already-existing moving image and the results of the determination at step S1103 are NO, first, the number of divisions of the update region of the new moving image is determined (step S1106). Following this, the flag “change in number of divisions” is set to the control region in the memory corresponding to the update region of the already-existing moving image (step S1107). After that, the divided update region that is transmitted at the current timing is determined (step S1116). Here, each divided update region is determined so that the divided update region of the new moving image is transmitted at the same time as the P-frame of the divided update region of the already-existing moving image. After that, the transmission timing determination processing is terminated. For the update region of the already-existing moving image, the number of divisions of the update region is set again at the time of the transmission of the I-frame of the update region of the already-existing moving image in the transmission processing of the moving image update region, to be described later (see steps S1306 to S1308 in
Next, in the case where the update region of a still image or the update region of a moving image whose transmission priority is not high has been detected in the update region priority determination processing illustrated in the flowchart in
After the moving image update region has not been detected and the results of the determination at step S1101 have changed to NO, whether or not the update region of a still image whose transmission priority is high has been detected is determined (step S1108).
In the case where the update region of a still image whose transmission priority is high has been detected and the results of the determination at step S1108 are YES, the transmission timing is adjusted as follows. The standby state continues until the current timing changes to the timing at which the P-frame of the moving image update region (or divided update region) whose transmission priority is high is transmitted (processing is repeated while the results of the determination at step S1109 are NO). In the case where the current timing has changed to the timing at which the P-frame of the moving image update region (or divided update region) whose transmission priority is high is transmitted and the results of the determination at step S1109 have changed to YES, the number of divisions of the update region of the new still image is determined (step S1110). After that, the divided update region (the update region itself if the number of divisions is 1) that is transmitted at the current timing is determined (step S1116). Here, in the case where there are moving image update regions (or divided update regions) or still-image update regions (or divided update regions) whose timing is the same and whose transmission priority is low, each divided update region is determined so that the data of another update region is transmitted before that of those update regions (or divided update regions). Also in the case where the still-image update region whose transmission priority is high, which is another update region, is divided and transmitted, each divided update region is determined so that the transmission of all the still image divided update regions is completed first. After that, the transmission timing determination processing is terminated.
In the case where the moving image update region has been detected and the results of the determination at step S1101 described previously have changed to YES, and further, the update region is the moving image update region whose transmission priority is low and the results of the determination at step S1102 have changed to NO, the transmission timing is adjusted as follows. The standby state continues until the current timing changes to the timing at which the P-frame of the moving image update region (or divided update region) whose transmission priority is high is transmitted (processing is repeated while the results of the determination at step S1111 are NO). Even in the case where the P-frame has been detected and the results of the determination at step S1111 have changed to YES, whether or not there is another update region whose transmission priority is high is determined, and if there is such an update region, the standby state continues until the timing at which the P-frame is transmitted is also reached for another update region (the results of the determination at step S1112 are NO). When the timing at which the P-frame is transmitted is reached for all the update regions whose transmission priority is high (the results of the determination at step S1112 are YES), the number of divisions of the new moving image update region is determined (step S1113). After that, the divided update region (the update region itself if the number of divisions is 1) that is transmitted at the current timing is determined (step S1116). Here, in the case where there are still-image update regions (or divided update regions) whose timing is the same and whose transmission priority is low, each divided update region is determined so that the data of another update region is transmitted before that of those update regions (or divided update regions). After that, the transmission timing determination processing is terminated.
In the case where the results of the determination at step S1101 and that at step S1108 described previously have changed to NO and another update region is the still-image update region whose transmission priority is low, the transmission timing is adjusted as follows. The standby state continues until the current timing changes to the timing at which the P-frame of the moving image update region (or divided update region) is transmitted (processing is repeated while the results of the determination at step S1114 are NO). The transmission priority of the P-frame may be high or low. In the case where the current timing has changed to the timing at which the P-frame of the moving image update region (or divided update region) is transmitted and the results of the determination at step S1114 have changed to YES, the number of divisions of the new still-image update region is determined (step S1115). After that, the divided update region (the update region itself if the number of divisions is 1) that is transmitted at the current timing is determined (step S1116). Here, even in the case where the transmission priority of the moving image update region is low, at the time of the transmission of the I-frame of the update region, the results of the determination at step S1114 will change to NO, and therefore, the data of the still-image update region whose transmission priority is low, which is another update region, is not transmitted. At the time of the transmission of the P-frame of the moving image update region, in the case where the still-image update region whose transmission priority is low and that is another update region can be transmitted at the same time as the transmission thereof, each divided update region is determined so that the transmission is performed. In the case where the still-image update region whose transmission priority is low and that is another update region will not be transmitted at the same time as the transmission of the moving image of the P-frame, each divided update region is determined so that the P-frame of the moving image update region and the still-image update region whose transmission priority is low and that is another update region are transmitted alternately.
Due to the functions of the update region priority determination unit 801 and the transmission timing determination unit 802 described above, or the update region priority determination processing in
In
The configuration in
By this control processing, even in the case where there is a plurality of update regions or where the moving image region has been changed, it is made possible to transmit the I-frames and the P-frames of the moving images of a plurality of divided update regions and the still images in the intermingled state at one timing.
First, the update region priority determination processing in
After that, whether or not the network band has changed (step S1302), whether or not the number of moving image update regions has changed (step S1303), and whether or not the “change in number of divisions” flag has been set (step S1304) are determined sequentially.
In the case where the results of any one of the determinations are YES, the number of divisions of the update region is set again as follows.
First, whether or not the current timing is the output timing of the moving image update region (step S1305) and whether or not the update region is the update region of the I-frame in the case where the update region is the moving image update region (step S1306) are determined.
In the case where the update region currently being subjected to the processing is the I-frame of the moving image and the results of the determinations at step S1305 and step S1306 are YES, the number of divisions of the update region is set again (step S1308) after the “change in number of division” flag is reset (step S1307).
After the processing at this step S1308 or after the results of all the determinations at step S1302, S1303, and S1304 have changed to NO, the division processing of the update region is performed (step S1310). Here, the I-frame and the P-frame of the moving image update region are divided so that the transmission timing of each divided update region explained in the transmission timing determination processing in
Then, the data of the divided update region that has been generated at step S1310 and that is transmitted at the current timing is transmitted to the client terminal 120 after being compressed as a moving image (step S1311).
After that, whether or not the transmitted data is the I-frame is determined (step S1312).
In the case where the transmitted data is the I-frame and the results of the determination at step S1312 are YES, the average compression ratio of the transmitted I-frame is estimated (step S1313).
In the case where the transmitted data is not the I-frame and the results of the determination at step S1312 are NO, the average compression ratio of the transmitted P-frame is estimated (step S1314).
After the processing at step S1313 or step S1314, the processing of the flowchart in
In the case where the current timing is not the output timing of the moving image update region and the results of the determination at step S1305 are NO, or in the case where the current moving image update region is not the I-frame and the results of the determination at step S1306 are NO, the following control processing is performed. The “change in number of divisions” flag is set so that the results of the determination at step S1304 will be YES at the next timing and steps S1305 to S1308 will be performed again (step S1309). After that, the processing proceeds to the processing at step S1310.
First, the update region priority determination processing in
After that, whether or not the network band has changed is determined (step S1402).
In the case where the network band has changed and the results of the determination at step S1402 are YES, the “change in number of divisions” flag is set (step S1403). As a result of this, at the timing at which the processing of the next moving image update region is performed, the following processing is performed. The number of divisions of the update region is set again at step S1308 because the results of the determination at step S1304 in
After the processing at step S1403 or after the results of the determination at step S1402 have changed to NO, the division processing of the update region is performed (step S1404). Here, the still-image update region is divided so that the transmission timing of each divided update region explained in the transmission timing determination processing in
Then, the data of the divided update region that has been generated at step S1404 and is transmitted at the current timing is transmitted to the client terminal 120 after being compressed as a still image (step S1405). After that, the processing of the flowchart in
The assumption about the moving image is the same as that in the case of the first embodiment described previously in
Here, the transmission time of the P-frame of the three divided moving image update regions is calculated as follows.
(15×8) [kbit]÷5,000 [kbps]=0.024 [sec]
The data amount of the still-image update region is assumed to be, for example, 135 [kB] at the time of compression. As a result of this, the transmission time to transmit the data of the still-image update region is calculated as an expression below.
(135×8) [kbit]÷5,000 [kbps]=0.216 [sec]
Consequently, in the case where the still-image update region and the P-frame are transmitted at the same time, transmitting the still image update region in a time period equal to or shorter than the time period that is calculated by an expression below is needed in order to complete transmission in 100 msec or less.
0.1 [sec]−0.024 [sec]=0.076 [sec]
As a result of this, it is not possible to transmit the still-image update region in one iteration, and therefore, the still-image update region is also divided. The number of divisions at this time is calculated by an expression below.
0.216÷0.076=2.84
By the above-described calculation expression, n≧2.84 holds, and therefore, the number of divisions is calculated as three. Consequently, as illustrated in
In the case where there is a plurality of moving image update regions, the number of divisions and the transmission timing are determined by the transmission timing determination unit 802 in accordance with the transmission priority determined by the update region priority determination unit 801.
If it is assumed that the priority of the update region 1601 and the update region 1602 is “1” (see
In the case where the data of the update region 1603 is transmitted at the time of the transmission of the P-frames of the update region 1601 and the update region 1602, the threshold value of the transmission time of the update region 1603 at the time of the transmission of the P-frames of the update region 1601 and the update region 1602 is 75.4 msec, and by the same calculation as that in the case of
For example, in the case where the network band has changed at a timing indicated by 1701 in
In the case where the network band has so narrowed that all the P-frames of the divided update regions of the update region can no longer be transmitted at the same time, the frame rate decreases even if the moving image update region is changed. However, the moving image update region is changed at the transmission timing of the next I-frame as follows by taking into consideration the transmission time of the I-frame.
It is not possible to transmit all of the P-frames at each transmission timing as indicated by 1703 or 1704 in
In
Immediately after the detection of a moving image update region whose transmission priority is high, the moving image compression of the new moving image update region is started.
The number of divisions is determined in the same manner as in the case of
At the timing at which the I-frame of the already-existing moving image update region is transmitted, for example at a timing indicated by 1802 in
In the case where a moving image update region whose transmission priority is low has been detected, as in the case of
In the case where the size of the update region is changed, the timing at which the size is changed is controlled.
In the case where a new update region is included within the current update region, at the transmission timing of the next I-frame, the new update region is changed into the moving image update region.
In the case where a new update region partially overlaps the current update region, a region that has not yet been changed into the current moving image update region is transmitted as a new moving image update region. The transmission as the new moving image update region is controlled similarly as in the case of
The computer illustrated in
The CPU 2001 controls the whole of the computer. The memory 2002 is a memory, such as a RAM, which temporarily stores a program or data stored in the external storage device 2005 (or the portable recording medium 2009) when the program is executed, or the data is updated, or the like. The CPU 2001 controls the whole of the computer by reading programs from the memory 2002 and executing the programs.
The input device 2003 detects an input operation by a user through a keyboard, a mouse, etc., and notifies the CPU 2001 of the detection results.
The output device 2004 outputs the data that is sent under the control of the CPU 2001 to a display device or a printing device.
The external storage device 2005 is, for example, a hard disk storage device, and is mainly used for saving various kinds of data and programs.
The portable recording medium drive device 2006 receives the portable recording medium 2009, such as an optical disc, an SDRAM, and a CompactFlash (registered trademark), and plays an auxiliary role in the external storage device 2005.
The communication interface 2007 is a device for connecting a communication line, such as, for example, a LAN (Local Area Network) and a WAN (Wide Area Network).
The system having the configuration in
All examples and conditional language provided herein are intended for the pedagogical purpose of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification related to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
Number | Date | Country | Kind |
---|---|---|---|
2014-023048 | Feb 2014 | JP | national |