Patent Grant
9781384
The present application claims the benefit of priority under 35 U.S.C. §119 of Japanese Patent Application No. 2015-218541, filed on Nov. 6, 2015, the contents of which are incorporated herein by reference in their entirety.
1. Field of the Invention
The disclosures discussed herein relate to an information processing system, an information processing apparatus, an information processing method, and a non-transitory computer-readable recording medium storing an information processing program.
2. Description of the Related Art
Related art technologies provide methods to conduct video conferences or TV conferences by mutually transmitting images and the like between multiple locations. Transmission of images and the like is frequently performed by encoding video data to transmit the encoded data.
When encoding schemes for encoding data differ between the terminals that attend the conference (hereinafter also called “conference attending terminals”) and a server that receives signals transmitted from the conference attending terminals, the server may frequently fail to decode the transmitted signals. Japanese Unexamined Patent Application Publication No. 2014-236472 discloses, for example, a following method to prevent the decoding of the signals from failing. In this method, an apparatus such as a gateway apparatus initially detects two encoding schemes, namely, a first encoding scheme used by the server for performing communications with the conference attending terminals and a second encoding scheme used by the conference attending terminals for performing communications with the server. The gateway apparatus subsequently determines whether the first encoding scheme matches the second encoding scheme. When the gateway apparatus determines that the first encoding scheme does not match the second encoding scheme, the gateway apparatus encodes signals received from the server with a second encoding scheme to transmit the encoded signals to the conference attending terminals, and also encodes signals received from the conference attending terminals with the first encoding scheme to transmit the encoded signals to the server. When the gateway apparatus determines that the first encoding scheme does not match the second encoding scheme, the gateway apparatus performs, instead of the conference attending terminals, negotiations between the server and the conference attending terminals.
An aspect of the present invention is directed to providing an information processing system capable of reducing process burdens for performing communications.
According to an embodiment, an information processing system includes a first information processing terminal and an information processing apparatus coupled to the first information processing terminal. The information processing system includes a storage configured to store codec data, the codec data including at least one first codec capable of being used for performing first communications between the first information processing terminal and a second information processing terminal differing from the first information processing terminal; a determiner configured to determine a second codec for performing second communications between the first information processing terminal, the second information processing terminal, and a third information processing terminal differing from the second information processing terminal, based on the codec data; and a switcher configured to switch a third codec currently used for performing the first communications, the third codec being selected from in the first codec.
Other objects, features and advantages of the present invention will become more apparent from the following detailed description when read in conjunction with the accompanying drawings.
The related art technology needs to have the server perform conversions or the like in order to conduct communications. The conversions may thus impose substantial processing-related burdens on the server.
The following illustrates preferred embodiments with reference the accompanying drawings. Note that duplicated descriptions may be omitted by assigning identical reference numbers to those components having substantially the same functional configurations in the specification and the drawings of the present application.
Overall Configuration Example
The information processing system 1 may be a transmission system configured to transmit content data and the like, a communications system configured to perform communications between multiple information processing terminals, or a combination of the transmission system and the communications system. Specifically, communications are performed by transmitting or receiving data indicating image data, voice data, or a combination of the image data and the voice data in the information processing system 1. Note that the images include static images moving images or combinations of the static and moving images. For example, the information processing system 1 may be a so-called a TV conference system, a TV telephone system, or the like. The following illustration is given based on an example of the TV conference system that serves as the information processing system 1.
In the following example of the TV conference system, an information processing terminal serving as a request source that transmits a request for starting a TV conference is called a “request source terminal”. Further, an information processing terminal serving as a request destination that receives the request for starting the TV conference is called a “destination terminal”.
The information processing system 1 illustrated in
In this example, it is assumed that the information processing terminal 10aa to an information processing terminal 10bc among multiple information processing terminals are installed in a first area AR1. On the other hand, it is assumed that an information processing terminal 10cc to the information processing terminal 10dc among the multiple information processing terminals are installed in a second area AR2. In the following illustration, any one of the information processing terminals 10aa to 10dc may be called an “information processing terminal 10”.
In this example, relay apparatuses 30a to 30d relays content data in order to transmit or receive the content data between the multiple information processing terminals. In the following illustration, any one of the relay apparatuses 30a to 30d may be called a “relay apparatus 30”.
In this example, a management system 50 serving as an example of an information processing apparatus is configured to perform so-called login authentication when a user of the terminal 10 performs a login operation. In addition, the management system 50 manages call statuses of the information processing terminals 10, a destination list, communications statuses of the relay apparatuses 30, and the like.
In this example, multiple routers 70a, 70b, 70c, 70d, 70ab, and 70cd are employed. In the following illustration, any one of the routers 70a, 70b, 70c, 70d, 70ab, and 70cd may be called a “router 70”. Each of the routers 70a, 70b, 70c, 70d, 70ab, and 70cd is configured to select an optimal route for transmitting and receiving data.
In this example, a program providing system 90 is employed. The program providing system 90 includes a storage device such as hard disk (HD) and the like. The program providing system 90 stores a terminal program to allow the terminal 10 to implement various types of functions or allow the information processing terminal 10 to serve as various types of components. The program providing system 90 may further be able to transmit the stored terminal program to the terminal 10. Similarly, the program providing system 90 stores a relay apparatus program to allow the relay apparatus 30 to implement various types of functions or allow the relay apparatus 30 to serve as various types of components. The program providing system 90 may further be able to transmit the stored relay apparatus program to the relay apparatus 30. In addition, the program providing system 90 stores a management system program to allow the management system 50 to implement various types of functions or allow the management system 50 to serve as various types of components. The program providing system 90 may further be able to transmit the stored management system program to the management system 50.
In this example, a maintenance system 100 is employed. The maintenance system 100 is a computer configured to perform maintenance, management, and repair on the terminals 10, the relay apparatuses 30, the management system 50, the program providing system 90, and the like. For example, it is assumed that the maintenance system 100 is installed domestically. On the other hand, it is assumed that the terminals 10, the relay apparatuses 30, the management system 50, and the program providing system 90 are installed internationally. In such an example, the maintenance system 100 performs perform maintenance, management, and repair on the terminals 10, the relay apparatuses 30, the management system 50, the program providing system 90, and the like via the communications network 2. The maintenance system 100 is a computer configured to perform maintenance including management of the model numbers, sales destinations, maintenance and inspection or failure histories of the terminals 10, the relay apparatuses 30, the management system 50, the program providing system 90, and the like.
As illustrated in
Meanwhile, the terminals 10ca to 10cc, the relay apparatus 30c, and the router 70c, and the like are coupled via a LAN 2c to perform communications with one another. Similarly, the terminals 10da to 10dc, the relay apparatus 30d, and the router 70d, and the like are coupled via a LAN 2d to perform communications with one another. The LAN 2c and LAN 2d are coupled via a dedicated line 2cd including a router 70cd to perform communications with one another. It is assumed that the LAN 2c and LAN 2d are constructed within a predetermined second area AR2. For example, the second area AR2 is the United States of America (USA). It is assumed that the LAN 2c is constructed within a New York (NY) Office and the LAN 2d is constructed with a Washington, D.C. Office. The first area AR1 and the second area AR2 are coupled via respective routers from the Internet 2i to perform communications with each other.
As illustrated in
Note that
The function of the terminals 10 is not limited to the communication function to talk over the telephones between the different offices or between the different rooms within the same office. The terminals 10 may be used as the communication function to talk over the telephones within the same room or between indoor and outdoor. Note that when the terminals 10 used as the communication function to talk over the telephones outdoor may perform communications with wireless communications such as mobile phone communications networks.
Initially, in the information processing system 1, a management information session sei is established between the request source terminal 10S and the destination terminal 10D for transmitting or receiving various types of management information DC via the management system 50.
Further, high resolution image data DH, intermediate resolution image data DM, low resolution image data DL, audio data DS, and the like are transmitted or received between the request source terminal 10S and the destination terminal 10D. Note that four sessions are established to transmit or receive data including the high resolution image data DH, intermediate resolution image data DM, low resolution image data DL, audio data DS, and the like. The four sessions may be called an image and audio data session sed in the following illustration. The image and audio data may be transmitted or received via the relay apparatus 30 by the image and audio data session sed, as illustrated in
Specifically, the image represented by the low resolution image data DL includes 160 by 120 pixels. The image represented by the intermediate resolution image data DM includes 320 by 240 pixels. The image represented by the high resolution image data DH includes 640 by 480 pixels.
For example, the low resolution image data DL are and received via the narrowband route. On the other hand, the low resolution image data DL and the intermediate resolution image data DM are transmitted and received via the relatively wideband route. In addition, the low resolution image data DL, the intermediate resolution image data DM, and the high resolution image data DH are transmitted and received via the wider band route.
Hardware Configuration Example
The CPU 101 includes processors configured to perform computation to implement processes and data processes performed by the terminal 10. The CPU 101 serves as control device configured to control the hardware components. The overall operations of the terminal 10 may thus be controlled by the CPU 101.
The ROM 102, the RAM 103, the flash memory 104, and the SSD 105 are examples of a storage. For example, the ROM 102 stores programs such as initial program loader (IPL) and the like for use in driving the CPU 101. The RAM 103 is an example of a main storage used as a work area of the CPU 101. Further, the SSD 105 is configured to store terminal programs and data including image data audio data in the flash memory 104, based on the control of the CPU 101.
The media drive 107 is configured to couple a medium 106 including recording media such as a flash memory and an optical disk to the terminal 10. The media drive 107 is configured to read data from or write data on the medium 106.
The operation button 108 is an example input device via which the user inputs operations. For example, the operation button 108 is used when the user selects a destination for the terminal 10 to perform communications with.
The power switch 109 is used to switch the power of the terminal 10 ON or OFF.
The network I/F 111 serves as an interface configured to couple the terminal 10 to the network. For example, the network N I/F 111 is configured to transmit to an external apparatus or receive from the external apparatus data via the communications network 2.
The camera 112 is configured to image a subject to generate image data of the subject. The camera 112 is controlled by the imaging element I/F 113. For example, the imaging element I/F 113 may transmit the image data generated by the camera 112 to the external apparatus via the communications network 2.
The microphone 114 is configured to input voice of a user to generate audio data of the user. The speaker 115 is configured to output voice based on the audio data. The audio input-output I/F 116 is configured to control the microphone 114 and the speaker 115.
The display I/F 117 is coupled to a display 120 via a cable 120c. The display 120 is an example of an output device configured to display images and icons or the like. The cable 120c may be an analog RGB (VGA) signal cable, a component video cable, high-definition multimedia interface (HDMI) (registered trademark) cable, a digital visual interface (DVI) cable, or the like.
The relay apparatus 30 includes a CPU 201, a ROM 202, a RAM 203, a hard disk (HD) 204, a hard disk drive (HDD) 205, a media drive 207, a display 208, and a network I/F 209. The relay apparatus 30 also includes a keyboard 211, a mouse 212, and a CD-ROM drive 214. The above-described hardware components are coupled via a bus line 210. The relay apparatus 30 thus serves as a computer.
The CPU 201 includes processors configured to perform computation to implement processes and data processes performed by the relay apparatus 30. The CPU 201 serves as control device configured to control the hardware components. The overall operations of the relay apparatus 30 may thus be controlled by the CPU 201.
The ROM 202, the RAM 203, the HD 204, and the HDD 205 are examples of a storage. For example, the ROM 202 stores programs such as an IPL and the like for use in driving the CPU 201. The RAM 203 is an example of a main storage used as a work area of the CPU 201. Further, the HDD 205 is configured to store relay apparatus programs and data including image data audio data in the HD 204, based on the control of the CPU 201.
The media drive 207 is configured to couple a medium 206 including recording media such as a flash memory and an optical disk to the relay apparatus 30. The media drive 207 is configured to read data from or write data on the medium 206.
The display 208 is an example of an output device configured to display images and icons or the like.
The network N I/F 209 serves as an interface configured to couple the relay apparatus 30 to the network. For example, the network N I/F 209 is configured to transmit to an external apparatus or receive from the external apparatus data via the communications network 2.
The keyboard 211 and a mouse 212 are an example input device via which the user inputs operations.
The CD-ROM drive 214 is configured to couple a medium 213 serving as a recording medium such as a CD-ROM to the relay apparatus 30. The CR-ROM drive 214 is configured to read data from or write data on the medium 213.
Functional Configuration Example
For example, the terminal 10 includes a transmitter-receiver 11, an operation input receiver 12, a login request part 13, an imager 14, a voice input part 15a, a voice output part 15b, a display controller 16, a final selector 17, a delay detector 18, a storing-reading processor 19, and a destination list creator 20. The terminal 10 further includes a nonvolatile storage 1001 and a volatile storage 1002.
The transmitter-receiver 11 is configured to transmit to an external apparatus or receive from the external apparatus data via the communications network 2. Note that the transmitter-receiver 11 may be implemented, for example, by the network I/F 111 (
The operation input receiver 12 is configured to receive various types of operations from the user. Note that the operation input receiver 12 may be implemented, for example, by the operation button 108 (
The login request part 13 is configured to request login when the power is switched ON. Note that the login request part 13 may be implemented, for example, by the network N T/F 111 (
The imager 14 is configured to image a subject such as a user, generate image data of the user, and output the generated image data of the user. Note that the imager 14 may be implemented, for example, by the camera 112 (
The voice input part 15a is configured to input sound and voice such as a user's voice to generate audio data. Note that the voice input part 15a may be implemented, for example, by the microphone 114 (
The voice output part 15b is configured to output voice based on the received audio data. Note that the voice output part 15b may be implemented, for example, by the speaker 115 (
The display controller 16 is configured to control displaying images on the display 120 (
The final selector 17 is configured to finally select one of the relay apparatuses among the multiple relay apparatuses. Note that the final selector 17 may be implemented, for example, by the CPU 101 and the like.
The delay detector 18 is configured to detect delay time (e.g., the unit system is ms) in receiving audio data and image data from the external apparatus. Note that the delay detector 18 may be implemented, for example, by the CPU 101 and the like.
The storing-reading processor 19 is configured to read data from or store data in the nonvolatile storage 1001 and the volatile storage 1002. Note that the storing-reading processor 19 may be implemented, for example, by the SSD 105 or the like. For example, the nonvolatile storage 1001 stores data including a terminal identifier (ID) and a password for identifying the terminal 10. The volatile storage 1002 stores image data and audio data received during communications. In this case, the image data and audio data are overwritten in the volatile storage 1002 every time the volatile storage 1002 receives the image data and audio data. The images or voice are output based on the image data and audio data stored in the volatile storage 1002.
The destination list creator 20 is configured to create and update a destination list indicating the destination candidate terminals based on a destination list and data including statuses of the destination terminal candidates received from the management system 50. Note that the destination list creator 20 may be implemented, for example, by the network I/F 111 and the like.
Note that each of the IDs including the terminal IDs and the relay apparatus IDs indicates characters, symbols, signs, graphics, numerical values, or a combination of the characters, the symbols, the signs, the graphics, and the numerical values.
The relay apparatus 30 includes a transmitter-receiver 31, a status detector 32, a data quality identifier 33, a change quality manager 34, a data quality changer 35, a reading-storing processor 39, and a nonvolatile storage 3000.
The nonvolatile storage 3000 includes a change quality management DB 3001 formed of a change quality management table (Table 1).
The change quality management table is configured to manage an IP address of a terminal indicating a relay destination of the image data and image quality of the image data relayed by the terminal.
The transmitter-receiver 31 is configured to transmit to an external apparatus or receive from the external apparatus data via the communications network 2. Note that the transmitter-receiver 31 may be implemented, for example, by the network I/F 209 (
The status detector 32 is configured to detect an operating status and the like of the relay apparatus 30. The operating status includes “online”, “offline”, “currently transmitting”, “temporarily stopped”, or the like. Note that the status detector 32 may be implemented, for example, by the CPU 201 (
The data quality identifier 33 is configured to search the change quality management DB 3001, for example, by the IP address of the destination terminal as a search key. This search extracts the image quality of the image data to be relayed to identify the extracted image quality. Note that the data quality identifier 33 may be implemented, for example, by the CPU 201 (
The print part 34 is configured to change the change quality management DB 3001 based on data indicating the quality received from the management system 50. For example, suppose that the conference is conducted between a request source terminal having the terminal ID “10aa” and a destination terminal having the terminal ID “10db” that mutually transmit or receive image data indicating high quality images. Note that the request source terminal is illustrated as the terminal 10aa, and the destination terminal is illustrated as the terminal 10db in
Suppose also that in addition to the above-described TV conference, another TV conference is conducted between the terminal 10bb serving as a request source terminal and the terminal 10ca serving as a destination terminal. Each of the bandwidths available for use in transmitting and receiving data in two TV conferences may become narrower than the bandwidths available for use in transmitting and receiving data in one TV conference. As a result, receiving the image data may be delayed in the terminal 10db, for example. In such a case, the relay apparatus 30 may handle the delay by lowering the image quality of the relaying image data. For example, the relay apparatus 30 that has been transmitting or receiving the image data with high image quality may transmit or receive the image data with the intermediate image quality. Further, when the relay apparatus 30 lowers the image quality, the relay apparatus 30 changes the change quality management DB 3001 based on the data indicating the quality of image data transmitted or received with the intermediate quality. Note that the print part 34 may be implemented, for example, by the CPU 201 (
The data quality changer 35 is configured to change an image quality of an image represented by the image data received from the request source terminal based on the change quality management DB 3001. Note that the data quality changer 35 may be implemented, for example, by the CPU 201 (
The storage-reader 39 is configured to read data from or store data in the nonvolatile storage 3000. Note that the storing-reading processor 39 may be implemented, for example, by the HDD 205 (
The management system 50 includes a transmitter-receiver 51, a terminal authenticator 52, a status manager 53, an extractor 54, a terminal status acquisition part 55, a primary selector 56, a session manager 57, a quality manager 58, a storing-reading processor 59, a delay time manager 60, and a nonvolatile storage 5000.
The nonvolatile storage 5000 includes a relay apparatus management DB 5001 formed of a relay apparatus management table (Table 2) noted below.
The relay apparatus management table is configured to manage an operating status of the relay apparatus, received date and time at which the management system 50 has received data indicating the operating status, an IP address of the relay apparatus, and the maximum data transmission rate of the relay apparatus, in association with the relay apparatus ID, for each of the relay apparatus IDs. For example, the relay apparatus having the relay apparatus ID “111a” indicates the operating status “online” in the relay apparatus management table. The received date and time, at which the management system 50 has received the data indicating the operating status of the relay apparatus 30 having the relay apparatus ID “111a”, indicate “Nov. 10, 2009 13:00”. The IP address of the relay apparatus 30 having the relay apparatus ID “111a” indicates “1.2.1.2”. The maximum data transmission rate of the relay apparatus 30 having the relay apparatus ID “111a” indicates “100 Mbps”.
The nonvolatile storage 5000 includes a terminal authentication management DB 5002 formed of a terminal authentication management table (Table 3) noted below.
The terminal authentication management table is configured to manage a password in association with a terminal ID, for each of the terminal IDs. For example, the terminal ID of the terminal 10aa (
The nonvolatile storage 5000 includes a terminal management DB 5003 formed of a terminal management table (Table 4) noted below.
The terminal management table is configured to manage a destination name, a terminal status, an IP address, a codec accessible by the terminal, and date and time at which the management system 50 has received data indicating these items in association with the terminal ID, for each of the terminal IDs. For example, the terminal ID of the terminal 10aa (
The nonvolatile storage 5000 includes a destination list management DB 5004 formed of a destination list management table (Table 5) noted below.
The destination list management table is configured to manage the terminal IDs of the destination terminals in the corresponding TV conference in association with the terminal ID of the request source terminal, for each of the request source terminals. For example, the request source terminal having the terminal ID “01aa” may determine the terminal having the terminal ID “01ab” as the destination terminal. The destination terminal candidates may thus be stored in association with the request source terminals in the destination list management table. The management system 50 that receives a request for adding or deleting the destination terminal candidates updates the destination list management table by adding terminals as destination terminal candidates or deleting the terminals from the destination list management table.
The nonvolatile storage 5000 includes a session management DB 5005 formed of a session management table (Table 6) noted below.
The session management table is configured to manage a session for selecting the relay apparatus in association with the selection session ID, for each of the selection session IDs. The session management table manages the relay apparatus ID, the terminal ID of the request source terminal, the terminal ID of the destination terminal, delay time, and received date and time at which the management system 50 has received data indicating these items. For example, the relay apparatus selected by the session having the selection session ID “se1” is stored in association with the relay apparatus ID “111a” in the session management table. The relay apparatus 30 having the relay apparatus ID “111a” relays the image data, the audio data, and the like between the request source terminal having the terminal ID “01aa” and the destination terminal having the terminal ID “01db”. The delay time of the data received by the destination terminal at “Nov. 10, 2009, 14:00” is “200 ms”. Note that in a case where the TV conference is conducted between the two terminals, the delay time and the like may be managed by the request source terminal instead of the destination terminal. However, in a case where the TV conference is conducted between the three or more terminals, the delay time is managed based on data transmitted from the received terminal.
The nonvolatile storage 5000 includes a priority management DB 5006 formed of an address priority management table (Table 7) and a transmission rate priority management table (Table 8) noted below.
The address priority management table is configured to manage the same or the difference of four decimal numbers forming the dot address in association with the address priority. Note that the IP address is a sequence of four decimal numbers separated by dots, and each decimal number indicates the binary value of one of four bytes. According to the address priority management table, the address priority becomes higher as the number of same decimal numbers (four address portions of the IP address) in the IP address increases. In the address priority management table illustrated in Table 7, “same” indicates the same decimal numbers (address portion) of the IP address whereas “different” indicates the different decimal numbers of the IP address. For example, when the three digits from the highest order digit of the dot address are the same, the address priority is “5”. When the two digits from the highest order digit of the dot address are the same, the address priority is “3”. When the highest order digit of the dot address is the same, the address priority is “1”. When the highest order digit of the dot address is different, the address priority is “0”. Note that in the address priority management table illustrated above (Table 7), “-” indicates the address portion unassociated with the address priority management.
The nonvolatile storage 5000 includes a quality management DB 5007 formed of a quality management table (Table 9) noted below.
The quality management table is configured to manage the delay time in association with the quality of the image data. According to the quality management table illustrated above (Table 9), the quality of images becomes lower as the delay time increases. Specifically, the relay apparatus degrades the quality of images as the delay time becomes longer.
The nonvolatile storage 5000 includes a quality management DB 5008 formed of a conference management table (Table 10) noted below.
The conference management table is configured to manage the “conference ID” assigned to each of the TV conferences in association with a “conference attending terminal ID” of each of the terminals that attends a conference, a “compatible codec” indicating codec(s) capable of being used by each of the conference attending terminals, a “common codec” indicating codec(s) capable of being used by all the conference attending terminals, and a “codec in use” indicating a codec currently used in the conference. The conference management table is updated every time a conference starts or an attendee of the conference attends or quits a conference.
Overall Process Example
In step S1, the first terminal 10A transmits a conference information request for requesting conference information to the management system 50 so that the management system 50 transmits the conference information including statuses of the registered conferences.
In step S2, the management system 50 acquires data indicating codecs from the terminal management DB 5003 via the extractor 54 (see
In step S3, the management system 50 transmits conference information and the like to the first terminal 10A.
In step S4, the first terminal 10A determines whether it is possible to perform a conference with another terminal, based on the received conference information and the like.
In step S5, the first terminal 10A displays a conference list and the like, based on a determined result and the like in step S4.
In step S6, the first terminal 10A inputs an operation of a user to select a location.
In step S7, the first terminal 10A transmits a conference request to the management system 50.
In step S8, the management system 50 transmits a conference request to the second terminal 10B, based on the received conference request of the first terminal 10A.
In step S9, the second terminal 10B approves the received conference request.
In step S10, the second terminal 10B that has approved the conference request transmits a conference start request for starting a conference with the first terminal 10A to the management system 50.
In step S11, the management system 50 updates the “codec in use” that is the codec used in the conference via the status manager 53 (
In step S12, the management system 50 starts the conference.
In step S21, the third terminal 10C transmits to the management system 50 a data request for displaying a conference list including conference information and the like presenting statuses of the registered conferences.
In step S22, the management system 50 acquires data relating to codecs including the “common codec”, the “codec in use”, and the like illustrated in Table 10 via the extractor 54 (
In step S23, the management system 50 transmits conference information and the like to the third terminal 10C.
In step S24, the third terminal 10C determines whether to perform a conference with other terminals, based on the conference information and the like.
In step S25, the third terminal 10C displays a conference list and the like, based on a determined result and the like in step S24.
In step S26, the third terminal 10C inputs an operation made by the user to select a conference.
In step S27, the third terminal 10C transmits a conference attending request for attending the conference to the management system 50.
In step S28, the management system 50 updates the “codec in use” determined based on the “common codec” capable of being used by all the information processing terminals that attend the conference.
In step S29, the management system 50 allows the third terminal 10C to attend the conference.
Note that the management system 50 may allow the information processing terminal to quit the conference after the overall process illustrated in
In step S31, the third terminal 10C inputs an operation performed by the user to select quitting the conference.
In step S32, the third terminal 10C transmits a conference quitting request for quitting the conference to the management system 50.
In step S33, the management system 50 causes the status manager 53 (see
In step S34, the management system 50 allows the third terminal 10C to quit the conference.
In addition, the following process may be performed by the relay apparatus 30 or the like after step S11 illustrated in
In steps S61-1 and S61-2, the management system 50 transmits preliminary relay request information illustrating indication to preliminarily request each of the relay apparatuses (30a and 30b) selected in the relay apparatus selecting process to perform a relay process before starting the TV conference. The preliminary relay request information includes a session ID “se1”, an IP address “1.2.1.3” of the request source terminal (the terminal 10aa) and an IP address “1.3.2.4” of the destination terminal (the terminal 10db). The preliminary relay request information allows the relay apparatuses (30a and 30b) to detect an appropriate one of the selection sessions, an appropriate terminal serving as the request source terminal, and an appropriate terminal serving as the destination terminal. In addition, the relay apparatuses (30a and 30b) may be able to detect the IP address “1.1.1.2” of the management system 50 serving as a transmission source of the preliminary relay request information.
In steps S62-1 and 62-2, the relay apparatuses (30a and 30b) transmit preliminary transmission request information from the respective transmitter-receivers 31 to the detected request source terminal (the terminal 10aa) via the communications network 2. The preliminary transmission request information includes indication to transmit the preliminary transmission information including a later-described “ping” (i.e., Packet Internet Groper). Similarly, the relay apparatuses (30a and 30b) also transmit the preliminary transmission request information to the relay apparatuses (30a and 30b) themselves before the TV conference starts. The preliminary transmission information includes the session ID “se1”. Based on the preliminary transmission information having the session ID “se1”, the request source terminal (the terminal 10aa) may be able to detect the transmission of the preliminary transmission information to the relay apparatuses (30a and 30b) in the process of selecting the relay apparatus 30 executed in the session of the session ID “se1”. Simultaneously, the request source terminal (the terminal 10aa) may be able to detect respective IP addresses (“1.2.1.2” and “1.2.2.2”) of the relay apparatuses (30a and 30b) serving as transmission sources of the preliminary transmission request information.
Note that the management system 50 does not directly report the IP address of the destination terminal (the terminal 10db) to the request source terminal (the terminal 10aa). The management system 50 may, for example, report the IP address of the destination terminal (the terminal 10db) to the relay apparatus 30a as illustrated in step S61-1. The management system 50 requests the relay apparatus 30b to transmit the preliminary transmission request information to the request source terminal (the terminal 10aa) as illustrated in step S61-2. Steps 61-1 and 61-2 are performed in order to ensure security by not allowing the terminals 10aa and 10db to detect the IP address of another terminal 10.
In steps S63-1 and 63-2, the request source terminal (the terminal 10aa) transmits the preliminary transmission information from the transmitter-receivers 31 to the respective relay apparatuses (30a and 30b) via the communications network 2. The preliminary transmission information is transmitted to the request source terminal (the terminal 10aa) via the respective relay apparatuses (30a and 30b) before transmission of the image data and the audio data.
The transmission of the preliminary transmission information enables measuring a time required between the transmission of the preliminary transmission information by the request source terminal (the terminal 10aa) and the reception of the preliminary transmission information by the destination terminal (the terminal 10db). The preliminary transmission information includes “ping” or pinging for ensuring coupling the request source terminal (the terminal 10aa), the relay apparatuses (30a and 30b), and the destination terminal (the terminal 10db) to mutually have communications. The preliminary transmission information further includes the session ID “se1” and the transmitted date and time at which the preliminary transmission information is transmitted from the request source terminal (the terminal 10aa).
The relay apparatuses (30a and 30b) may be able to detect that the preliminary transmission information is transmitted by the execution of the session having the selection session ID “se1”. Simultaneously, the relay apparatuses (30a and 30b) may be able to detect the IP address “1.2.1.3” of the request source terminal (the terminal 10aa) serving as a transmission source of the preliminary transmission information.
In steps S64-1 and S64-2, the relay apparatuses (30a and 30b) relay the preliminary transmission information to the IP address of the destination terminal (the terminal 10db) included in the received preliminary relay request information. The destination terminal (the terminal 10db) may be able to detect that the preliminary transmission information is transmitted by the execution of the session having the selection session ID “se1”. Simultaneously, the destination terminal (the terminal 10db) may be able to detect the IP addresses (“1.2.1.2” and “1.2.2.2”) of the relay apparatuses (30a and 30b) serving as transmission sources (relaying sources) of the preliminary transmission information.
In step S65, the final selector 17 of the destination terminal (the terminal 10db) finally selects one of the relay apparatuses 30 to relay image data and audio data in the TV conference based on the preliminary transmission information.
In step S65-1, a measuring part of the final selector 17 detects received date and time at which the preliminary transmission information relayed from each of the relay apparatuses 30 is received by a receiver of the destination terminal (the terminal 10db) (see
In step S65-2, a calculating part of the final selector 17 calculates time required between the transmission and the reception of the preliminary transmission information based on the difference between the reception time and the transmission time of the preliminary transmission information included in the corresponding preliminary transmission information, for each preliminary transmission information.
In step S65-3, a final selector determines whether all the preliminary transmission information pieces, the number of which corresponds to the number of relay apparatuses of “2” that relay the respective preliminary transmission information pieces, are received in the execution of the session having the session ID “se1”.
In step S65-4, when the final selector determines that not all the preliminary transmission information pieces have been received (NO in step S65-3), the final selector determines whether a predetermined time (one minute in this example) has elapsed from the reception of the preliminary transmission information by the destination terminal (the terminal 10db). When the final selector determines that the predetermined time has not elapsed from the reception of the preliminary transmission information (NO in step S65-4), the final selector returns to step S65-1. When the final selector determines that all the preliminary transmission information pieces are received (YES in step S65-3) or that the predetermined time has elapsed from the reception of the preliminary transmission information (YES in step S65-4), the final selector proceeds with step S65-5.
In step S65-5, the final selector selects one of the relay apparatuses that has performed a relaying process within a shortest required time of all the required times. This embodiment represents an example of selecting the relay apparatus 30a, based on the assumption in which the required time for the preliminary transmission information relayed by the relay apparatus 30a is shorter than the required time for the preliminary transmission information relayed by the relay apparatus 30b.
Referring back to
In step S67, the session manager 57 of the management system 50 stores and manages the relay apparatus ID “111a” of the selected relay apparatus 30a. Note that the relay apparatus ID is stored in a relay apparatus ID field of the record having the session ID “se1” in the session management table of the session management DB 5005 (
In step S68, the transmitter-receiver 51 of the management system 50 transmits relay start request information indicating a relay start request for starting the relay to the relay apparatus 30a via the communications network 2. The relay start request information include the IP address “1.2.1.3” of the request source terminal (the terminal 10aa) to be relayed and the TP address “1.3.2.4” of the destination terminal (the terminal 10db).
In step S69, the relay apparatus 30a establishes a session for communicating image data of three different resolutions (low, intermediate and high resolutions) and audio data between the terminals 10aa and 10db. Having established the session, the terminals 10aa and 120db may be able to start a TV conference.
Note that the transmission source and the reception source may be switched between the request source terminal (the terminal 10aa) and destination terminal (the terminal 10db) after the relay apparatus selecting information has been transmitted by the management system 50 to the request source terminal (the terminal 10aa). The request source terminal (the terminal 10aa) may thus be able to perform the relay apparatus selecting process and transmission of selection information.
The following illustrates a process performed in the process of allowing a new information processing terminal to attend the conference currently in progress illustrated in
In step S41, the third terminal 10C transmits a conference start request corresponding to the conference ID “Conf04” to the management system 50. Note that the third terminal 10C may also transmit the conference start request using the codec “H264” in step S41.
In step S42, the management system 50 performs a determining and switching process.
In step S42-1, the information processing system determines whether the management system 50 has received the conference start request. For example, when step S41 illustrated in
In step S42-2, the information processing system acquires “compatible codec(s)”. For example, the information processing system acquires a codec or codecs recorded in the “compatible codec” field illustrated in Table 10. In this example, the information processing system acquires the compatible codecs “H264” and “H265” input into the “compatible codec” field corresponding to the conference TD “Conf04” in Table 10.
In step S42-3, the information processing system determines whether there is a “common codec” that is a codec common between the conference attending terminals. Specifically, the information processing system determines whether the compatible codecs acquired in step S42-2 include a compatible codec that is capable of being used by the terminal that has transmitted the conference start request (the third terminal 10C in this example). In this example, the acquired compatible codecs corresponding to the conference ID “Conf04” are “H264” and “H265” (see Table 10), and the compatible codec for the third terminal 10C (
When the information processing system determines that there is a “common codec” (YES in step S42-3), the information processing system subsequently proceeds with step S42-4.
On the other hand, when the information processing system determines that there is no common codec (NO in step S42-3), the information processing system proceeds with step S42-6. In step S42-6, the information processing system displays an error screen or the like.
In step S42-4, the information processing system determines whether the “common codec” differs from the “codec in use” (see Table 10). Specifically, in step S42-4, the information processing system compares the codec capable of being used by the third terminal 10C and the codec currently used by the terminals (terminal IDs 01da and 01ba in this case) in the conference having the conference ID “Conf04” (see Table 10), and determines whether the codec capable of being used by the third terminal 10C matches the codec currently used in the conference having the conference ID “Conf04” (e.g., “H265” in “codec in use” corresponding to “Conf04” in Table 10).
Referring back to
When the information processing system determines that the “common codec” does not differ from the “codec in use”; that is, when the information processing system determines that the “common codec” is the same as the “codec in use”, information processing system allows the third terminal 10C to attend the conference using the “codec in use” currently used by the other terminals in the conference.
On the other hand, when the information processing system determines that the “common codec” differs from the “codec in use”, the information processing system proceeds with step S42-5.
In step S42-5, the information processing system switches the “codec in use” currently used by the other terminals into the codec capable of being used by the third terminal 10C.
In step S42-5-1, the management system 50 transmits a codec switching request for switching a codec. In this example, the management system 50 transmits the codec switching request to each of the first terminal 10A and the second terminal 10B to switch the codec from “H265” to “H264”.
In step S42-5-2, each of the first terminal 10A and the second terminal 10B transmits a response to the codec switching request. Specifically, in step S42-5-2, the first terminal 10A and the second terminal 10B determine whether to use the requested codec (“H264”), and subsequently transmit respective responses based on the determined results. That is, when each of the first terminal 10A and the second terminal 10B is able to use the requested codec (“H264” in this example), the response is “OK”. On the other hand, when each of the first terminal 10A and the second terminal 10B is not able to use the requested codec, the response is “NG”.
In step S42-5-3, when the management system 50 has received the response “OK” from all the terminals (the first terminal 10A and the second terminal 10B in this example), the management system 50 deletes an entry and the like having the conference ID “Conf04” in Table 10. Next, the management system 50 generates a new conference ID and updates codec data including the “common codec”, “codec in use”, and the like corresponding to the new conference ID in Table 10.
In step S42-5-4, the management system 50 manages the relay apparatuses. Specifically, the management system 50 updates the session management DB 5005 to start a new TV conference.
In step S42-5-5, the management system 50 establishes a session. In this example, the information processing system establishes a new session for conducting a TV conference using a requested codec “H264”. Note that the screen and the audio data corresponding to the conference ID “Conf04” are output to the first terminal 10A and the second terminal 10B at this point.
In step S42-5-6, the first terminal 10A and the second terminal 10B that have received the image data and the like switch the screen to a screen for the newly established session.
In step S42-5-7, each of the first terminal 10A and the second terminal 10B transmits a quitting request for quitting the conference having the conference ID “Conf04” to the management system 50.
In step S42-5-8, the management system 50 transmits a response to the quitting request from each of the first terminal 10A and the second terminal 10B.
When the management system 50 receives the quitting request for quitting the conference having the conference ID “Conf04” from each of the first terminal 10A and the second terminal 10B, the management system 50 transmits a conference stop request to the relay apparatus 30 to stop the conference having the conference ID “Conf04”. On the other hand, the first terminal 10A, the second terminal 10B, and the third terminal 10C are able to continue the conference based on the newly established session.
Note that the quitting process is not limited to the quitting process in which a quitting request is transmitted each of from the first terminal 10A and the second terminal 10B described above. For example, the quitting request may be transmitted from the management system 50 to each of the first terminal 10A and the second terminal 10B.
Note that two sessions are established in a period between the time at which the new session is established and the time at which the session is switched to the newly established session. This status in which two sessions are established may be so called “double sessions” or the like. The “double sessions” may hereinafter be called “double sessions WS”. In this example, the double sessions WS include the already established session using the codec “H265” and the newly established session using the codec “H264”. In the case of the double sessions WS, it may be preferable to transmit or receive data and the like via the already established session. In this example, it may be desirable to perform communications with the session using the codec “H265” while the double sessions WS are established.
Moreover, it may be desirable to stop transmitting or receiving data with the session using the codec “H264” while the double sessions WS are established.
For example, to transmit or receive data with both the sessions of the double sessions WS using codecs “H264” and “H265”, two sets of data compatible to “H264” and “H265” need to be generated for the two sessions of the double sessions WS. Specifically, two sets of data transmitted or received in the double sessions WS are differently encoded by the CPU of each of the terminals, and hence, differently encoded two sets of the data are generated. In this example, different sets of data are generated for being adapted to the codecs “H265” and “H264”. As a result, each of the terminals uses the CPU processing power to perform two different encodings. On the other hand, when one of the sessions transmits or receives the data and the like while the other one of the sessions stops transmitting or receiving the data, only one encoding is required. Thus, the CPU processing power to be used may be reduced.
When the data and the like are transmitted or received in both the double sessions WS, a large amount of data may be transmitted or received, thereby narrowing the remaining bandwidths. On the other hand, when the data and the like are transmitted or received via one of the double sessions WS while the data and the like stop being transmitted or received via the other one of the double sessions WS, the amount of data transmitted or received may be reduced, compared to the amount of data transmitted or received via both of the double sessions WS. Hence, when the data and the like are transmitted or received via one of the double sessions WS while the data and the like stop being transmitted or received via the other one of the double sessions WS, the remaining bandwidths may be prevented from being narrowed.
A newly established session may be composed of partially established parts between the double sessions WS. Hence, when the newly established session includes partially unestablished parts, the video images or the like in communications may be interrupted (disconnected) while the conference is in progress. On the other hand, when the data are transmitted or received via the already established session, the information processing system may be able to prevent video images or the like in communications from being interrupted, compared to the case in which the data are transmitted or received via the newly established session.
Accordingly, the information processing system may be able to change the conference using the codec “H265” conducted between the first terminal 10A and second terminal 10B into the conference using the codec “H264” conducted between the first terminal 10A, the second terminal 10B, and the third terminal 10C.
Note that the determining and switching process is not limited to the example illustrated in
In step S42-10, the information processing system acquires a requested codec transmitted in step S41 along with the conference start request.
In step S42-11, the information processing system determines whether the requested codec is included in the “common codec(s)”. Specifically, the information processing system determines whether the requested codec is included in the “common codecs” acquired in step S42-2. In the following illustration, it is assumed that the acquired codecs (the acquired common codecs) are “H264” and “H265”, and the requested codec is “H264”. Hence, the information processing system determines that there is a “common codec” (YES in step S42-11) in this example. Specifically, the information processing system compares a common codec commonly used by all the terminals that attend a conference with the requested codec.
Subsequently, when the information processing system determines that there is a requested codec (YES in step S42-11), the information processing system proceeds with step S42-12. On the other hand, when the information processing system determines that there is no requested codec (NO in step S42-11), the information processing system proceeds with step S42-13.
In step S42-12, the information processing system determines whether the requested codec differs from the “codec in use”. Specifically, in step S42-12, the information processing system compares the requested codec and the “codec in use” that is the codec currently used by all the conference attending terminals in the conference having the conference ID “Conf04”, and determines whether the requested codec matches the “codec in use” currently used in the conference having the conference ID “Conf04”.
When the information processing system determines that the requested codec differs from the “codec in use” (YES in step S42-12), the information processing system proceeds with step S42-5 in
In step S42-13, the information processing system transmits a response to the conference start request. For example, the information processing system returns an error presenting failing to start conference as a response to the conference start request. Receiving such a response to the conference start request, the third terminal 10C may display an error screen and the like depicted in
The information processing system is unable to start a conference when the requested codec and the “common codec” are compared, and the requested codec is not included in the compatible codecs; that is, the information processing system is unable to start a conference unless the requested codec matches one of the “common codecs”. When the information processing system is unable to start a conference due to the mismatch between the requested codec and one of the “common codecs”, the error screen and the like may thus be displayed. On the other hand, when the requested codec is included in the “common codec(s)”, the requested codec is compared with the “codec in use”. Subsequently, when the requested codec matches the “codec in use” that is the codec currently used by all the conference attending terminals in the conference in progress, the information processing system allows the third terminal 10C to participate in the conference using the “codec in use” currently used in the conference.
On the other hand, when the requested codec does not match the “codec in use” that is the codec currently used in the conference in progress, the information processing system switches the “codec in use” currently used in the conference into the requested codec in the switching process illustrated in
Outline
In the information processing system according to the embodiment, communications are performed between the multiple information processing terminals by causing the information processing terminals to mutually transmit and receive data such as image data, audio data, or combination of the image data and the audio data between the information processing terminals. For example, it is assumed that communications are performed between a first terminal serving as an example of the first information processing terminal and a second terminal serving as an example of the second information processing terminal. The communications performed between the first terminal and the second terminal are called “first communications”. The following illustrates an example on the basis of the assumption in which the first communications are performed.
The first communications are performed using a codec (hereinafter called “first codec”) capable of being used by the first terminal and the second terminal to perform communications. For example, the first codec is determined based on the “compatible codec(s)” for each of the terminals illustrated in Table 4. In this example, the first codec is determined as a codec common between the “compatible codec” of the first terminal and the “compatible codec” of the second terminal. As already noted above, the codec common between the first terminal and the second terminal is called a “common codec”. The first codec is stored as the “common codec” or the like illustrated in Table 10. The information processing system may be able to detect the first codec based on the data (hereinafter called “codec data”) including information associated with codecs such as “common codec(s)”, “compatible codec(s)”, “codec(s) in use” and the like in Table 10.
Next, it is assumed that the third terminal serving as an example of the third information processing apparatus participates in the first communications already being performed. Communications performed between the first terminal, the second terminal, and a newly attending third terminal that has participated in the first communications already being performed are called “second communications”. The following illustrates an example of performing the second communications.
The second communications need to be performed using a codec capable of being used by all the first terminal, the second terminal, and the third terminal. The codec for use in performing the second communications is called a “second codec”. The codec currently used in the first communications is called a “third codec”. For example, the “third codec” indicates the “codec in use” in Table 10 that is the codec currently used in the first communications selected from the “common codec” in Table 10. The codec capable of being used by the third terminal alone is called a “fourth codec”.
Initially, the third codec is specified by the “codec in use” and the like in Table 10. That is, the third codec is specified by the codec data. The fourth codec is specified by the “compatible codec” and the like corresponding to the third terminal in Table 4. For example, in step S42-2 of the process illustrated in
Subsequently, the process includes comparing the “common codec” and the fourth codec that is capable of being used by the third terminal alone in step S42-3 illustrated in
The information processing system initially compares the first codec (the “common codec”) capable of being used by the first terminal and the second terminal with the fourth codec capable of being used by the third terminal. When the fourth codec matches the first codec (the “common codec”), the information processing system determines that the third terminal may participate in the first communications already being performed between the first terminal and the second terminal. Subsequently, the information processing system compares the third codec (the “codec in use”) currently used by both the first and the second terminals in the first communications and the fourth codec capable of being used by the third terminal. When the third codec (the “codec in use”) differs from the fourth codec being used by the third terminal, the information processing system switches the third codec into the fourth codec used as the second codec that is capable of being used by all the first, the second, and the third terminals. Accordingly, whereas the third codec is used in the first communications between the first and the second terminals, the fourth codec is used as the second codec for performing the second communications between the first, the second, and the third terminals. The information processing system may thus enable conducting a TV conference or the like by switching the third codec into the fourth codec so as to allow all the first, the second, and the third terminals to attend the TV conference or the like.
Compared to a case in which the information processing apparatus performs different encodings on data being transmitted to or received from the different terminals, the information processing system according to the embodiment performs communications using the second codec (i.e., the “common codec” used by all the conference attending terminals). The information processing system is thus capable of reducing process burdens caused by the use of different encodings and the like. The information processing system according to the embodiment may be able to reduce process burdens for performing communications.
There is a case where the third terminal specifies a requested codec to newly attend the conference. That is, the second communications are directed to be performed with the requested codec. In this case, the information processing system initially compares the first codec with the requested codec as illustrated in
Subsequently, the information processing system compares the currently used third codec and the requested codec. When the compared result indicates that the third codec differs from the requested codec, the information processing system switches the currently used third codec into the requested codec. The third codec is used in the first communications whereas the requested codec is used as the second codec used in the second communications. The information processing system thus enables conducting a TV conference or the like by causing all the first, the second, and the third terminals to attend the TV conference or the like with the codec desired by the user of the third terminal.
Further, various types of processing may include redundant processing, decentralized processing, and parallel processing, or combinations of the redundant processing, the decentralized processing, and the parallel processing.
Note that the above-described embodiments may be implemented by a program for causing a computer such as an information processing system having an information processing terminal, an information processing apparatus, or one or more information processing apparatuses to execute an information processing method.
The information processing system according to the above-described embodiments may be able to reduce process burdens for performing communications.
Note that definitions of terms used in the specification and claims are noted below.
The first codec is a “common codec” that indicates a “compatible codec” commonly used between the first terminal and the second terminal.
The second codec is a “codec in use” that is used in the second communications.
The third codec is a “codec in use” that is currently used in the first communications.
The fourth codec is a “compatible codec” specific to the third terminal.
The codec data indicate data including the first codec, the second codec, the third coded, the fourth coded, and the like.
The first communications indicate communications performed between the first terminal and the second terminal using the third codec.
The second communications indicate communications performed between the first terminal, the second terminal, and the third terminal using the second codec.
The preferred embodiments of the present invention are described above. However, the present invention is not limited to the specifically disclosed embodiments, and variations, alterations and modifications may be made without departing from the gist of the embodiment recited in the scope of the present invention.
The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present invention may be implemented as computer software implemented by one or more networked processing apparatuses. The network can comprise any conventional terrestrial or wireless communications network, such as the Internet. The processing apparatuses can compromise any suitably programmed apparatuses such as a general purpose computer, personal digital assistant, mobile telephone (such as a WAP or 3G-compliant phone) and so on. Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any storage medium for storing processor readable code such as a floppy disk, hard disk, CD ROM, magnetic tape device or solid state memory device.
The hardware platform includes any desired kind of hardware resources including, for example, a central processing unit (CPU), a random access memory (RAM), and a hard disk drive (HDD). The CPU may be implemented by any desired kind of any desired number of processor. The RAM may be implemented by any desired kind of volatile or non-volatile memory. The HDD may be implemented by any desired kind of non-volatile memory capable of storing a large amount of data. The hardware resources may additionally include an input device, an output device, or a network device, depending on the type of the apparatus. Alternatively, the HDD may be provided outside of the apparatus as long as the HDD is accessible. In this example, the CPU, such as a cache memory of the CPU, and the RAM may function as a physical memory or a primary memory of the apparatus, while the HDD may function as a secondary memory of the apparatus.
The present invention is not limited to the specifically disclosed embodiments, and variations and modifications may be made without departing from the scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2015-218541 | Nov 2015 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 8713148 | Imai et al. | Apr 2014 | B2 |
| 8850184 | Imai | Sep 2014 | B2 |
| 9210371 | Morita et al. | Dec 2015 | B2 |
| 9374556 | Nagamine et al. | Jun 2016 | B2 |
| 20030210659 | Chu | Nov 2003 | A1 |
| 20140118475 | Nagamine et al. | May 2014 | A1 |
| 20140354767 | Imai | Dec 2014 | A1 |
| 20140362718 | Nagamine et al. | Dec 2014 | A1 |
| 20140368410 | Imai et al. | Dec 2014 | A1 |
| 20150365727 | Nagamine et al. | Dec 2015 | A1 |
| 20160080449 | Nagamine et al. | Mar 2016 | A1 |
| 20160094594 | Imai et al. | Mar 2016 | A1 |
| 20160094595 | Shiro et al. | Mar 2016 | A1 |
| 20160112675 | Morita et al. | Apr 2016 | A1 |
| 20160277462 | Morita et al. | Sep 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 2014-236472 | Dec 2014 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20170134697 A1 | May 2017 | US |
