Patent Application
20230100767
This application is based on and claims priority under 35 USC 119 from Japanese Patent Application No. 2021-156014 filed Sep. 24, 2021.
The present disclosure relates to an information processing device, an information processing method, and a non-transitory computer readable medium.
In a service such as an online conferencing system in which at least speech is exchanged among multiple users, a separate conversation may take place between specific users among the multiple users in some cases.
Japanese Unexamined Patent Application Publication No. 6-164741 describes a system that divides multiple participants into multiple groups in advance, and achieves communication within the groups and communication for the conference as a whole.
Japanese Unexamined Patent Application Publication No. 2015-046822 describes a device that enhances and reproduces the voice of a specific participant.
In some cases, it may be desirable to have a separate conversation take place between specific users, without causing the specific users to stop participating in a conversation taking place among all of the multiple users.
Aspects of non-limiting embodiments of the present disclosure relate to a service in which at least speech is exchanged among multiple users such that a conversation takes place among all of the multiple users, and provides a mechanism that makes it possible for a separate conversation to take place between specific users without causing the specific users to stop participating in the conversation taking place among all of the multiple users.
Aspects of certain non-limiting embodiments of the present disclosure address the features discussed above and/or other features not described above. However, aspects of the non-limiting embodiments are not required to address the above features, and aspects of the non-limiting embodiments of the present disclosure may not address features described above.
According to an aspect of the present disclosure, there is provided an information processing device including a processor configured to output, in a case where a service is being used in which at least speech is exchanged among multiple users such that a conversation takes places among all of the multiple users, a speech of a separate conversation distinctly from a speech of the conversation taking place among all of the multiple users to a device of a user who is engaged in the separate conversation with a specific user from among the multiple users, and output the speech of the conversation taking place among all of the multiple users without outputting the speech of the separate conversation to a device of a user who is not engaged in the separate conversation.
An exemplary embodiment of the present disclosure will be described in detail based on the following figures, wherein:
An information processing system according to an exemplary embodiment will be described with reference to
The information processing system according to the exemplary embodiment includes an online conferencing system 10 and N terminal devices (where N is an integer equal to or greater than 1), for example. In the example illustrated in
The online conferencing system 10 and the terminal devices 12 have a function of communicating with other devices. The communication may be wired communication using a cable, or wireless communication. The wireless communication is a technology such as short-range wireless communication or Wi-Fi (registered trademark). The short-range wireless communication is a technology such as Bluetooth (registered trademark) or radio-frequency identifier (RFID), for example. Each device may also communicate with another device through a communication channel such as a local area network (LAN) or the Internet.
The online conferencing system 10 provides a service allowing the exchange of information among multiple users.
The information exchanged in the service is, for example, images, video, audio, text, signs and symbols other than text, files, or a combination of at least two of the above. Obviously, information other than the above may also be exchanged. Exchanging information refers to transmitting information and receiving information.
The service is a service provided by an online conference, for example. In the online conference, information such as speech, images, and video for example is exchanged among multiple users. An online conference is also referred to as a web conference, a remote conference, or a video conference. The service may also be a service that provides a social networking service (SNS). In the following, the service provided by the online conferencing system 10 will be referred to as the “online conferencing service”. Although the term “conference” is included in the name of the service out of convenience, the service may also be used for purposes other than a conference. In such cases, information such as speech, images, and video for example is likewise exchanged among multiple users.
Each terminal device 12 is a device such as a personal computer (hereinafter referred to as a “PC”), a tablet PC, a smartphone, or a mobile phone, for example.
A user uses the terminal device 12 to access the online conferencing system 10 and use the online conferencing service provided by the online conferencing system 10. For example, by having multiple users use their own respective terminal devices 12 to use the online conferencing service, information is exchanged among the multiple users.
In the online conferencing service, information is exchanged among multiple terminal devices 12, for example. A user account for using the online conferencing service may be created for each user, and information may be exchanged among multiple user accounts.
For example, an address (for example, a URL) for accessing and using the online conferencing service is generated by the online conferencing system 10. Each user acquires and accesses the address using his or her own terminal device 12, thereby enabling the user to use the online conferencing service corresponding to the address.
For example, if a user who acts as the host of an online conference uses his or her own terminal device 12 to request the online conferencing system 10 for use of the online conferencing service, an address for accessing the online conferencing service is generated by the online conferencing system 10. As a response to the request from the user, the address is transmitted from the online conferencing system 10 to the terminal device 12. It is assumed that the user acquiring the address will transmit the address to other users who will participate in the same online conferencing service. With this arrangement, each user is able to acquire the address to access and participate in the same online conferencing service.
Channels may also be created for the online conferencing service, and information may be exchanged among multiple users in each channel. For example, the online conferencing system 10 generates, for each channel, an address for accessing and using the online conferencing service corresponding to the channel. By accessing one of the addresses using the terminal device 12, a user is able to use the online conferencing service in the channel corresponding to the accessed address.
A service ID and a corresponding password for using the online conferencing service may also be generated by the online conferencing system 10, and the online conferencing service corresponding to the service ID and the password may be provided to users. A user acquires the service ID and corresponding password, uses the terminal device 12 to access the online conferencing system 10, and transmits the service ID and the password to the online conferencing system 10. With this arrangement, the online conferencing service corresponding to the service ID and the password is provided to the user.
A channel service ID and a corresponding password may also be generated for each channel by the online conferencing system 10, and the online conferencing service in the channel corresponding to the service ID and the password may be provided to users.
In the online conferencing service according to the exemplary embodiment, at least speech is exchanged among the multiple users participating in the same online conferencing service, and a conversation takes place among all of the multiple users. Additionally, a separate conversation also takes place between specific users from among the multiple users. For example, a conversation takes place among all of the multiple users participating in the online conferencing service in the same channel. In addition, a separate conversation takes place between specific users from among the multiple users participating in the online conferencing service in the same channel.
Hereinafter, the conversation that takes place among all of the multiple users participating in the same online conferencing service is referred to as the “overall conversation”, and the conversation that takes place separately between specific users is referred to as the “separate conversation”.
In the overall conversation, information is exchanged among all of the multiple users participating in the same online conferencing service. For example, speech uttered by a certain user in the online conferencing service is outputted to all other users participating in the online conferencing service. The speech is emitted from a speaker used by each user.
In the separate conversation, the information (speech, for example) exchanged between the specific users is outputted only to the specific users, and is not outputted to users not participating in the separate conversation.
In the online conferencing service, at least speech is exchanged among the multiple users such that the overall conversation takes place among all of the multiple users. With respect to the terminal devices 12 of the users participating in the separate conversation, the speech of the separate conversation and the speech of the overall conversation are outputted distinctly. With respect to the terminal devices of the user not participating in the separate conversation, the overall speech of the multiple users (that is, the speech being exchanged in the overall conversation) is outputted, but the speech of the separate conversation is not outputted.
In the overall conversation and the separate conversation, information other than speech, such as images, video, or text, may also be exchanged, and the information such as images, video, or text may be displayed on a display of each user's terminal device 12.
Note that the online conferencing system 10 corresponds to one example of an information processing device. Some of the processes executed by the online conferencing system 10 may also be executed by the terminal device 12.
Hereinafter,
The online conferencing system 10 includes a communication device 14, a user interface (UI) 16, a memory 18, and a processor 20, for example.
The communication device 14 is a communication interface including components such as a communication chip or a communication circuit, and has a function of transmitting information to another device and a function of receiving information from another device. The communication device 14 may have a wireless communication function, and may also have a wired communication function.
The UI 16 is a user interface, and includes a display and an operation device. The display is a liquid crystal display (LCD), an electroluminescence (OLED) display, or the like. The operation device is a device such as a keyboard, a mouse, input keys, or a control panel. The UI 16 may also be a UI such as a touch panel combining a display with an operation device. The UI 16 may also include a microphone and a speaker.
The memory 18 is a device that establishes one or multiple storage areas that store data. For example, the memory 18 is a hard disk drive (HDD), a solid-state drive (SSD), any of various types of memory (such as RAM, DRAM, or ROM, for example), another type of storage device (such as an optical disc, for example), or a combination of the above. One or multiple memories 18 are included in the online conferencing system 10.
The processor 20 is configured to control the operation of each unit of the online conferencing system 10. The processor 20 may include a memory.
Hereinafter,
The terminal device 12 includes a communication device 22, a UI 24, a memory 26, and a processor 28, for example.
The communication device 22 is a communication interface including components such as a communication chip or a communication circuit, and has a function of transmitting information to another device and a function of receiving information transmitted from another device. The communication device 22 may have a wireless communication function, and may also have a wired communication function.
The UI 24 is a user interface, and includes a display, an operation device, a microphone, a speaker, and a camera. The display is a liquid crystal display (LCD), an electroluminescence (OLED) display, or the like. The operation device is a device such as a keyboard, a mouse, input keys, or a control panel. The UI 24 may also be a UI such as a touch panel combining a display with an operation device.
The memory 26 is a device that establishes one or multiple storage areas that store data. For example, the memory 26 is a hard disk drive (HDD), a solid-state drive (SSD), any of various types of memory (such as RAM, DRAM, or ROM, for example), another type of storage device (such as an optical disc, for example), or a combination of the above. One or multiple memories 26 are included in each terminal device 12.
The processor 28 is configured to control the operation of each component of each terminal device 12. The processor 28 may also include a memory.
Hereinafter,
In the following, the users participating in the overall conversation but not participating in the separate conversation are referred to as the “conference participants”, and the specific users participating in the separate conversation are referred to as the “separate talkers”. The separate conversation occurs between multiple separate talkers participating in the overall conversation. In other words, the separate talkers participate in the separate conversation while also participating in the overall conversation.
In the example illustrated in
The online conferencing system 10 includes a system base unit 30, a separate conversation unit 32, and a communication control unit 34. These units are achieved by the processor 20 of the online conferencing system 10.
The system base unit 30 has a function of achieving the overall conversation. Specifically, the system base unit 30 transmits and receives, through a communication channel, information such as speech, images, and video with respect to the terminal devices 12 (for example, the terminal devices 12A, 12B, 12C, 12D, and 12E) of all users participating in the same online conferencing service. In other words, through the communication channel, the system base unit 30 receives information such as speech, images, and video from the terminal devices 12 of all users participating in the same online conferencing service, and transmits information such as speech, images, and video to the terminal devices 12 of all users participating in the same online conferencing service. For example, the system base unit 30 receives information (speech, for example) transmitted from the terminal device 12A, and transmits the information to the terminal devices 12B, 12C, 12D, and 12E. In this way, in the overall conversation, information such as speech, images, and video is shared by all users participating in the overall conversation.
The separate conversation unit 32 has a function of achieving the separate conversation. Specifically, the separate conversation unit 32 transmits and receives, through a communication channel, information such as speech, images, and video with respect to the terminal devices 12 of the multiple specific users engaged in the separate conversation. In other words, through the communication channel, the separate conversation unit 32 receives information such as speech, images, and video from the terminal device 12 of a specific user engaged in the separate conversation, and transmits the received information to the terminal device 12 of another specific user engaged in the separate conversation. The separate conversation unit 32 also manages the requesting and accepting of separate conversations.
For example, the separate conversation unit 32 receives information (speech, for example) transmitted from the terminal device 12D of the separate talker 1, and transmits the information to the terminal device 12E of the separate talker 2. Similarly, the separate conversation unit 32 receives information transmitted from the terminal device 12E of the separate talker 2, and transmits the information to the terminal device 12D of the separate talker 1.
The communication control unit 34 has a speech control function, a microphone control function, and an image processing function, and controls processes such as the output of speech, the output from a microphone, sound pickup by a microphone, and the processing of images generated by image capture performed by a camera.
The speech control function controls the output of the speech of the overall conversation and the speech of the separate conversation according to the functions of a speaker (for example, a speaker in the terminal device 12) used by the separate talker engaged in the separate conversation in the online conferencing service.
If the speaker used in the online conferencing service by a separate talker engaged in the separate conversation (for example, the speaker in the separate talker's terminal device 12) is a stereo speaker, the speech control function outputs the speech of the separate conversation and the speech of the overall conversation from respectively different channels of the stereo speaker.
If the speaker used in the online conferencing service by a separate talker engaged in the separate conversation (for example, the speaker in the separate talker's terminal device 12) is a monaural speaker, the speech control function raises the volume of the speech of the separate conversation higher than the volume of the speech of the overall conversation, and outputs the speech of the overall conversation and the speech of the separate conversation from the monaural speaker. If the speech of the separate conversation is silent, the speech control function outputs the speech of the overall conversation at a normal volume (that is, without changing the volume of the speech of the overall conversation).
The microphone control function controls output and sound pickup by a microphone (for example, a microphone in the terminal device 12) used by the separate talker engaged in the separate conversation in the online conferencing service.
Specifically, the microphone control function sets the output destination of sound picked up by a microphone used by a separate talker engaged in the separate conversation (for example, a microphone in the separate talker's terminal device 12) to only the terminal device 12 used by another separate talker engaged in the same separate conversation. In other words, during a separate conversation, the microphone control function mutes the output to the overall conversation from the microphone used by a separate talker, and sets the output destination of sound picked up by the microphone to only a separate talker.
In the case where the separate talker is asked for a response from the conference participants participating in the overall conversation, the microphone control function sets the output destination of the microphone used by the separate talker to the overall conversation while a “response handling operation” is ongoing during the separate conversation. If the “response handling operation” ends, the microphone control function sets the output destination of the microphone used by the separate talker to only the terminal device 12 used by another separate talker engaged in the same separate conversation. The “response handling operation” will be described in detail later.
The image processing function is a function executed in the case where images of users participating in the online conferencing service are transmitted and received in the online conferencing service. Specifically, in the case where an image is captured by a camera used by the separate talker (for example, a camera in the separate talker's terminal device 12) in the online conferencing service, the image processing function processes the image generated by the image capture performed by the camera. For example, the image processing functions alters the mouth of the separate talker appearing in the image to generate an image in which the mouth does not appear to be moving.
The communication control unit 34 transmits the processed image to the terminal devices 12 of the conference participants participating in the overall conversation but not participating in the separate conversation, and causes the processed image to be displayed on the display of each terminal device 12 of the conference participants. The communication control unit 34 transmits the unprocessed real image to the terminal device 12 of the separate talker engaged in the separate conversation, and causes the unprocessed image to be displayed on the display of the separate talker's terminal device 12.
The “received data” and the “data for overall conversation” illustrated in
The “transmitted data” represents the information (such as speech, images, and video, for example) transmitted to the overall conversation by the terminal devices 12 of the conference participants participating in the overall conversation.
The “received data for separate conversation” is the information (such as speech, images, and video, for example) exchanged in the separate conversation, and is the information received by the terminal devices 12 of the separate talkers participating in the separate conversation. The received data for separate conversation is only shared with the separate talkers, and is not shared with users other than the separate talkers.
The “transmitted data for separate conversation” is the information (such as speech, images, and video, for example) transmitted to the separate conversation by the terminal devices 12 of the separate talkers engaged in the separate conversation.
Note that all or part of the speech control function, the microphone control function, and the image processing function may also be achieved by the processor 28 of each terminal device 12. Additionally, some or all of the functions of the separate conversation unit 32 may also be achieved by the processor 28 of each terminal device 12.
Hereinafter,
The separate conversation unit 32 monitors the terminal device 12 of each user participating in the overall conversation, and if a separate conversation request from a terminal device 12 is detected (S01, Yes), the separate conversation unit 32 calls the receiver on the other end of the separate conversation, and asks the receiver for confirmation about whether to accept or refuse the separate conversation (S02). The receiver is a user participating in the overall conversation. For example, if a certain user uses his or her own terminal device 12 to specify the other end of a separate conversation and request the online conferencing system 10 to execute a separate conversation, the separate conversation unit 32 receives the request and transmits query information about whether or not to start the separate conversation to the terminal device 12 of the receiver on the other end of the separate conversation. Hereinafter, the user who requests the separate conversation may also be referred to as the “initiator”.
The receiver receives the query about whether or not to start the separate conversation, and uses his or her own terminal device 12 to accept or refuse the separate conversation (S03). For example, information indicating the query is displayed on the display of the receiver's terminal device 12, and the receiver responds to the query. Information indicating acceptance or refusal is transmitted from the receiver's terminal device 12 to the online conferencing system 10.
In the case where the receiver accepts the separate conversation (S03, Accepted), the separate conversation unit 32 calls the communication control unit 34 (S04).
In the case where the receiver refuses the separate conversation (S03, Refused), the process ends. Also, in the case where a separate conversation request is not detected (S01, No), the process ends.
Hereinafter,
If the separate conversation is accepted as described above, the separate conversation unit 32 creates a channel for the separate conversation between the terminal device 12 of the conference participant who requested the separate conversation (in other words, the initiator) and the terminal device 12 of the conference participant who received the separate conversation request and accepted the separate conversation (in other words, the receiver) (S11). With this arrangement, the conference participant who requested the separate conversation and the conference participant who accepted the separate conversation engage in a separate conversation as separate talkers.
The speech control function of the communication control unit 34 checks the functions of the speaker used by the separate talker who engages in the separate conversation, and controls the output of the speech of the overall conversation and the speech of the separate conversation according to the functions (S12).
The microphone control function of the communication control unit 34 sets the output destination of the microphone used by the separate talker engaged in the separate conversation to the channel for the separate conversation, and sets the output destination of the microphone to the overall conversation while the response handling operation is ongoing (S13).
In the case where an image is being captured by the camera used by the separate talker, the image processing function of the communication control unit 34 processes an image of the separate talker's face outputted to the overall conversation (S14).
If the separate conversation has not ended (S15, No), the process returns to step S12. If the separate conversation has ended (S15, Yes), the separate conversation unit 32 disconnects the channel for the separate conversation (S16). The communication control unit 34 reverts the settings of the speaker and the microphone used by the separate talker to the settings from before the separate conversation.
Hereinafter,
First, the speech control function of the communication control unit 34 checks the type of the speaker used by the separate talker (for example, the speaker in the terminal device 12 used by the separate talker) (S21).
If the speaker used by the separate talker is a monaural speaker (S21, Monaural), the speech control function controls the output of speech according to the monaural speaker type (S22).
In a situation without a separate conversation interruption (S23, No), the speech control function outputs the speech of the overall conversation (for example, the conversation of the conference as a whole) at a normal volume (that is, without changing the volume of the speech of the overall conversation) (S24). For example, if the speech of the separate conversation is silent, the speech control function outputs the speech of the overall conversation at a normal volume.
In the case of detecting a separate conversation interruption (S23, Yes), the speech control function lowers the volume of the speech of the overall conversation (for example, the conversation of the conference as a whole), raises the volume of the speech of the separate conversation higher than the volume of the speech of the overall conversation, and outputs the speech of the overall conversation and the speech of the separate conversation from the monaural speaker (S25). In this way, the speech control function prioritizes the output of the speech of the separate conversation over the output of the speech of the overall conversation. In the case of detecting that the separate conversation is silent, the speech control function revers the volume of the speech of the overall conversation to a normal volume.
If the separate conversation has not ended (S26, No), the process returns to step S23 and the speech control function controls the output of speech according to the monaural speaker type. If the separate conversation has ended (S26, Yes), the processes by the speech control function end.
If the speaker used by the separate talker is a stereo speaker (S21, Stereo), the speech control function controls the output of speech according to the stereo speaker type (S27). Namely, the speech control function outputs the speech of the separate conversation and the speech of the overall conversation from respectively separate channels in the stereo speaker. For example, the speech control function outputs the speech of the separate conversation from one channel in the stereo speaker and outputs the speech of the overall conversation from another channel in the stereo speaker, and thereby outputs the speech of the separate conversation and the speech of the overall conversation separately.
If the separate conversation has not ended (S28, No), the speech control function controls the output of speech according to the stereo speaker type. If the separate conversation has ended (S28, Yes), the processes by the speech control function end.
The speech control function checks the type of speaker used by each of the initiator and the receiver described above, and controls the output of speech from each speaker according to the monaural speaker method or the stereo speaker method.
The microphone control function sets the output destination of sound picked up by the microphone used by the separate talker (for example, the microphone in the separate talker's terminal device 12) to only the terminal device 12 used by another separate talker engaged in the same separate conversation. For example, the microphone control function sets the output destination of sound picked up by the microphone used by the initiator to only the receiver's terminal device 12, and sets the output destination of sound picked up by the microphone used by the receiver to only the initiator's terminal device 12. With this arrangement, the speech of the separate conversation is exchanged only between the separate talkers.
In the case where the “response handling operation” is not being performed during the separate conversation (S31, No), the processes related to the “response handling operation” end.
In the case where the processes of the “response handling operation” are set to active during the separate conversation (S31, Yes), the microphone control function causes a warning to be displayed on the display of the terminal devices 12 used by each of the initiator and the receiver (S32). For example, warning information indicating that the separate conversation has been suspended is displayed on each display.
While the processes of the “response handling operation” are set to active, the microphone control function switches the output destination of the microphone used by the separate talker performing the “response handling operation” to the overall conversation (for example, the conversation of the conference as a whole) (S33). For example, the “response handling operation” functions only while a specific key or on-screen button is being pressed, and does not function if the specific key or on-screen button is not being pressed.
If the “response handling operation” ends, the microphone control function sets the output destination of the microphone used by the separate talker to only the terminal device 12 used by another separate talker engaged in the same separate conversation.
The image processing function checks whether or not the camera used by the separate talker (such as an inward-facing camera installed in the separate talker's terminal device 12, for example) is active (S41). For example, the camera is set to active in the case where the camera is powered on and a setting that outputs an image generated by image capture performed by the camera to the online conferencing service is on. The camera is set to inactive in the case where the camera is powered off or the setting that outputs the image to the online conferencing service is off.
In the case where the camera is not active (S41, No), the processes by the image processing function end.
In the case where the camera is active (S41, Yes), if the “response handling operation” is being performed (S42, Yes), the process returns to step S41. In this case, the image generated by the image capture performed by the camera is transmitted to the terminal devices 12 of the other conference participants without being altered by the image processing function, and is displayed on the display in the terminal devices 12 of the other conference participants. For example, in the case where an image of the separate talker's face is captured by the camera, the real image is displayed on the display in the terminal devices 12 of the other conference participants without altering the image of the face.
In the case where the camera is active (S41, Yes), if the “response handling operation” is not being performed (S42, No), the image processing function duplicates a video for the overall conversation (that is, for the conversation of the conference as a whole) (S43). In other words, the image processing function duplicates a video generated by the image capture performed by the camera used by the separate talker in real time for the overall conversation (S43).
Next, the image processing function specifies the mouth area of the face from the duplicated video, and sets the specified mouth area as a processing area (S44).
Next, the image processing function culls images of open mouths from the duplicated video (S45).
Next, the image processing function combines the video (S46). Specifically, the image processing function combines the video by interpolating images of the mouth area on the basis of a preceding image (that is, an image of a closed mouth) preceding the image of the open mouth and a succeeding image (that is, an image of a closed mouth) succeeding the image of the open mouth. In this way, the image processing function generates a video in which the mouth is not open.
Next, the communication control unit 34 outputs the processed video as a video image of the separate talker him- or herself to the overall conversation (that is, the conversation of the conference as a whole) (S47). In other words, the communication control unit 34 transmits the processed video to the terminal devices 12 of the conference participants participating in the overall conversation but not participating in the separate conversation, and causes the processed video to be displayed on the display of each terminal device 12 of the conference participants. The communication control unit 34 transmits the unprocessed real video to the terminal device 12 of the separate talker engaged in the separate conversation, and causes the unprocessed video to be displayed on the display of the separate talker's terminal device 12. After that, the process returns to step S41.
A video 36 is a video generated by image capture performed by the camera used by a certain separate talker, and is a video before culling. The video 36 contains multiple frames (that is, images). In the example illustrated in
In the video 36, the frames 36b and 36e illustrate an open mouth, while the frames 36a, 36c, 36d, 36f, and 36g illustrate a closed mouth.
A video 38 is obtained by culling the frames 36b and 36e of an open mouth from the video 36.
A video 40 is a video obtained after performing interpolation. The frame 36a preceding the frame 36b is inserted at the position of the culled frame 36b as a frame 40b, and the frame 36d preceding the frame 36e is inserted at the position of the culled frame 36e as a frame 40e. The frames 36a and 36d are frames of a closed mouth. By replacing the frame at the position where the frame 36b was located with the frame 36a of a closed mouth, a frame is interpolated at the position where the frame 36b was located. Similarly, by replacing the frame at the position where the frame 36e was located with the frame 36d of a closed mouth, a frame is interpolated at the position where the frame 36e was located. Through such interpolation, the video 36 is processed to generate the video 40. Note that the mouth area, an open mouth, and a closed mouth are specified by using known image processing technology.
The communication control unit 34 transmits the processed video 40 as a video of the separate talker him- or herself to the terminal devices 12 of the conference participants participating in the overall conversation but not participating in the separate conversation, and causes the processed video 40 to be displayed on the display of each terminal device 12 of the conference participants. The communication control unit 34 transmits the unprocessed real video 36 to the terminal device 12 of the separate talker engaged in the separate conversation, and causes the unprocessed video to be displayed on the display of the separate talker's terminal device 12.
Hereinafter, a specific example of the exemplary embodiment will be described with reference to
Here, as an example, five users (for example, conference participants 1 to 3 and separate talkers 1, 2) are participating in the same online conferencing service. The conference participants 1 to 3 are participating in the overall conversation but are not engaged in the separate conversation. The separate talkers 1, 2 are engaged in the separate conversation while also participating in the overall conversation. The separate talker 1 is the initiator who requests the separate talker 2 for the separate conversation, and the separate talker 2 is the receiver who receives the separate conversation request from the separate talker 1.
For example, the conference participant 1 uses the terminal device 12A to participate in the online conferencing service, the conference participant 2 uses the terminal device 12B to participate in the online conferencing service, the conference participant 3 uses the terminal device 12C to participate in the online conferencing service, the separate talker 1 uses the terminal device 12D to participate in the online conferencing service, and the separate talker 2 uses the terminal device 12E to participate in the online conferencing service.
Hereinafter,
On the screen 42D, images representing the users participating in the same online conferencing service are displayed. An image 44A is an image representing the conference participant 1, and is generated by image capture performed by the camera in the terminal device 12A. An image 44B is an image representing the conference participant 2, and is generated by image capture performed by the camera in the terminal device 12B. An image 44C is an image representing the conference participant 3, and is generated by image capture performed by the camera in the terminal device 12C. An image 44D is an image representing the separate talker 1, and is generated by image capture performed by the camera in the terminal device 12D. An image 44E is an image representing the separate talker 2, and is generated by image capture performed by the camera in the terminal device 12E. Note that the images 44A to 44E may also be video images. Here, the images 44A to 44E are assumed to include video.
The images (that is, the images 44A, 44B, 44C) of the conference participants 1, 2, 3 not engaged in the separate conversation are generated by image capture performed by the respective cameras, and are unprocessed real images. In the case where the separate conversation is not taking place, the images (that is, the images 44D, 44E) of the separate talkers 1, 2 are generated by image capture performed by the respective cameras, and are unprocessed real images. In this way, images representing each of the participants are shared by all of the conference participants 1 to 3 and the separate talkers 1, 2.
The speech of each participant is outputted to the overall conversation, that is, to all other participants. For example, in the case where the separate conversation is not taking place, the speech of the separate talker 1 is outputted to the conference participants 1 to 3 and the separate talker 2, and is emitted from the respective speakers of the conference participants 1 to 3 and the separate talker 2. The same applies to the other participants. In this way, the speech of each of the participants is shared by all of the conference participants 1 to 3 and the separate talkers 1, 2.
Also, objects (such as documents and images, for example) displayed on the screen are displayed on the respective displays of the terminal devices 12 of the conference participants 1 to 3 and the separate talkers 1, 2, and are shared by all of the conference participants 1 to 3 and the separate talkers 1, 2.
If the separate talker 1 uses his or her own terminal device 12D to specify the separate talker 2 as the other end of the separate conversation on the screen 42D (for example, if the separate talker 1 clicks, touches, or taps the image 44E of the separate talker 2), the processor 20 of the online conferencing system 10 displays a menu 46 on the screen 42D. On the menu 46, a button for requesting the separate conversation, a button for ending the separate conversation, a button for chat, a button for email, and the like are displayed.
If the separate talker 1 selects and presses the “Request separate conversation” button from the menu 46 (for example, if the separate talker 1 clicks, touches, or taps the button), the separate conversation unit 32 causes information indicating that the separate talker 1 is requesting a separate conversation with the separate talker 2 to be displayed on the display in the terminal device 12E used by the separate talker 2. If the separate talker 2 uses the terminal device 12E to give an instruction for accepting the separate conversation with the separate talker 1 in response to the request, the separate conversation unit 32 receives the acceptance of the separate conversation and creates a channel for the separate conversation between the separate talker 1 and the separate talker 2. Thereafter, the processes from steps S12 to S15 illustrated in
In the case where the separate talker 2 refuses the separate conversation with the separate talker 1, the separate conversation unit 32 does not create a channel for the separate conversation between the separate talker 1 and the separate talker 2.
Note that in some cases, a certain participant (for example, the separate talker 2) may receive separate conversation requests from multiple different participants. In this case, the participant receiving the requests may select a participant to engage in a separate conversation with from among the multiple different participants and accept the separate conversation with the selected participant, or refuse all of the requests from the multiple different participants.
Additionally, a separate conversation may also take place among three or more participants. For example, while a separate conversation is taking placed between the separate talkers 1 and 2, the separate talker 1 or the separate talker 2 may request another participant to join the same separate conversation taking place between the separate talkers 1 and 2. If the other participant accepts the request to join, the separate conversation takes places among the separate talkers 1, 2 and the other participant. In this case, if a separate talker (for example, the separate talker 2) other than the separate talker (for example, the separate talker 1) who requested the other participant to join accepts the participation by the other participant, the other participant is allowed to join the separate conversation.
A button 48 and a button 50 are also displayed on the screen 42D. The button 48 is for leaving the online conferencing service. If the button 48 is pressed, the participant pressing the button 48 leaves the online conferencing service. The button 50 is for responding. Responding will be described later.
In the case where a separate conversation is taking place between the separate talkers 1 and 2, the separate conversation unit 32 causes information (text and an image, for example) indicating that the separate talkers 1 and 2 are engaged in the separate conversation to be displayed only on the display in the terminal device 12D used by the separate talker 1 and the terminal device 12E used by the separate talker 2. For example, as indicated by the sign 52, a line connecting the image 44D with the image 44E and text indicating that a separate conversation is taking place are displayed on the screen 42D. Similar information is also displayed on the screen displayed with respect to the separate talker 2. Information indicating that the separate talkers 1 and 2 are engaged in a separate conversation is not displayed on the respective displays in the terminal devices 12 of the conference participants 1 to 3.
During the separate conversation, the output of speech is controlled by the processes illustrated in
Processed images of the separate talkers 1 and 2 are transmitted to the respective terminal devices 12 (that is, the terminal devices 12A, 12B, and 12C) of the conference participants 1 to 3, and the processed images of the separate talkers 1 and 2 are displayed on the respective displays in the terminal devices 12A, 12B, and 12C. For example, the images 44D and 44E displayed on the respective displays of the terminal devices 12A, 12B, and 12C are images that have been processed by the image processing function of the communication control unit 34. As described with reference to
Real images of the separate talkers 1 and 2 are transmitted to the respective terminal devices 12 (that is, the terminal devices 12D and 12E) of the separate talkers 1 and 2, and the real images of the separate talkers 1 and 2 are displayed on the respective displays of the terminal devices 12D and 12E. For example, the images 44D and 44E displayed on the respective displays of the terminal devices 12D and 12E are real images generated by image capture. In other words, the image 44D displayed on the respective displays of the terminal devices 12D and 12E is a real image generated by image capture performed by the camera in the terminal device 12D. The same applies to the image 44E.
Real images of each of the conference participants 1 to 3 are displayed on the respective displays of the terminal devices 12A, 12B, 12C, 12D, and 12E. In other words, the images 44A, 44B, and 44C displayed on the respective displays of the terminal devices 12A, 12B, 12C, 12D, and 12E are real images generated by image capture performed by the cameras.
The speech of each of the conference participants 1 to 3 (in
The speech of the separate talker 1 (in
The overall conversation (that is, the conversation of the conference as a whole) is emitted from the respective speakers in the terminal devices 12 (that is, the terminal devices 12A, 12B, and 12C) of the conference participants 1 to 3.
The speaker in the terminal device 12D of the separate talker 1 is a stereo speaker, and therefore speech is outputted from the speaker in the terminal device 12D according to the stereo speaker method (see
The speaker in the terminal device 12E of the separate talker 2 is a monaural speaker, and therefore speech is outputted from the speaker in the terminal device 12E according to the monaural speaker method (see
Note that a screen for the separate conversation may also be displayed on the displays in the terminal devices 12 of the separate talkers and shared by only the separate talkers. For example, a screen for the separate conversation is displayed only on the respective displays in the terminal devices 12D and 12E, and information displayed on the screen for the separate conversation is shared by only the separate talkers 1 and 2.
Hereinafter,
As described above, the communication control unit 34 outputs the speech of the separate talker (for example, the separate talker 1) engaged in a separate conversation only to the terminal device 12 (for example, the terminal device 12E) of the other separate talker (for example, the separate talker 2) engaged in the same separate conversation. In this case, when the separate talker (for example, the separate talker 1) responds to the overall conversation, the communication control unit 34 suspends the separate conversation and outputs the speech of the separate talker to the overall conversation. For example, the speech of the separate talker 1 is outputted to the respective terminal devices 12 of the conference participants 1 to 3 and the separate talker 2, and is emitted from the speaker in each terminal device 12.
The communication control unit 34 causes an image for suspending the separate conversation and responding to the overall conversation to be displayed on the display in the terminal device 12 of the separate talker. The communication control unit 34 may also cause the image to be displayed on the displays of the terminal devices 12 of all participants participating in the online conferencing service. If the image is operated by the separate talker, the communication control unit 34 suspends the separate conversation and outputs the speech of the separate talker to the overall conversation.
The button 50 for responding is one example of an image for suspending the separate conversation and responding to the separate conversation. The communication control unit 34 outputs the speech of the separate talker to the overall conversation while the button 50 is being pressed by the separate talker, and outputs the speech of the separate talker only to the other end of the separate conversation when the button 50 is not being pressed.
Additionally, in the case where the separate talker responds to the overall conversation, or in other words, in the case where the button 50 is being pressed, the communication control unit 34 treats an unprocessed image of the separate talker (that is, a real image generated by image capture performed by the camera in the terminal device 12 used by the separate talker) as an image representing the separate talker, and causes the image to be displayed on the displays in the terminal devices 12 of the other participants.
For example, suppose that while the separate talkers 1 and 2 are engaged in the separate conversation, a conference participant (for example, the conference participant 1) calls out to the separate talker 1 (by, for example, saying something like “I'd like to hear Separate Talker 1's opinion”). The conference participant 1 calls out to the separate talker 1 through the overall conversation. Since the speaker of the separate talker 1 is a stereo speaker and the speech of the overall conversation is emitted from one channel, the separate talker 1 is able to recognize that the conference participant 1 is calling out to the separate talker 1.
To respond to being called out, the separate talker 1 presses the button 50 on the screen 42D. While the button 50 is being pressed, the communication control unit 34 switches the output destination of the microphone used by the separate talker 1 to the overall conversation. With this arrangement, utterances by the separate talker 1 are outputted to the overall conversation, or in other words to the respective terminal devices 12 (that is, the terminal devices 12A, 12B, 12C, and 12E) of the conference participants 1 to 3 and the separate talker 2, and are emitted from the respective speakers in the terminal devices 12A, 12B, 12C, and 12E. In this case, as illustrated in
Also, while the separate talker 1 is pressing the button 50, the communication control unit 34 causes the real image 44D (that is, the image 44D not processed by the image processing function) generated by image capture performed by the camera in the terminal device 12D used by the separate talker 1 to be displayed on the respective displays in the terminal devices 12 (that is, the terminal devices 12A, 12B, 12C, 12D, and 12E) of the participants. Similarly, the communication control unit 34 causes the real image 44E (that is, the image 44E not processed by the image processing function) generated by image capture performed by the camera in the terminal device 12E used by the separate talker 2 on the other end of the separate conversation to be displayed on the respective displays in the terminal devices 12 of the participants.
When the separate talker 1 releases the button 50 (that is, in the case where the button 50 is not being pressed), the communication control unit 34 switches the output destination of the microphone used by the separate talker 1 to the separate conversation with the separate talker 2. Similarly, the communication control unit 34 switches the output destination of the microphone used by the separate talker 2 to the separate conversation. The communication control unit 34 also causes the images 44D and 44E processed by the image processing function to be displayed on the respective displays in the terminal devices 12A, 12B, and 12C, and causes the unprocessed images 44D and 44E to be displayed on the respective displays in the terminal devices 12D and 12E.
During a separate conversation, a conference participant not participating in the separate conversation may also request a separate conversation with a separate talker. In this case, the separate talker receiving the request may end the separate conversation and start a new separate conversation with the requesting conference participant.
Note that the processor 20 of the online conferencing service may also convert the content of the conversation into text and cause the converted text to be displayed on the display in the terminal device 12 of each participant. The conversion may also be performed by the processor 28 in the terminal device 12 of each participant.
The functions of the online conferencing system 10 and the terminal device 12 above are achieved by the cooperative action of hardware and software as an example. For instance, the functions of each device are achieved by causing a processor in each device to load and execute a program stored in a memory of each device. The program is stored in the memory through a recording medium such as a CD or DVD, or alternatively through a communication channel such as a network.
In the embodiments above, the term “processor” refers to hardware in a broad sense. Examples of the processor include general processors (e.g., CPU: Central Processing Unit) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, and programmable logic device). In the embodiments above, the term “processor” is broad enough to encompass one processor or plural processors in collaboration which are located physically apart from each other but may work cooperatively. The order of operations of the processor is not limited to one described in the embodiments above, and may be changed.
The foregoing description of the exemplary embodiments of the present disclosure has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Obviously, many modifications and variations will be apparent to practitioners skilled in the art. The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical applications, thereby enabling others skilled in the art to understand the disclosure for various embodiments and with the various modifications as are suited to the particular use contemplated. It is intended that the scope of the disclosure be defined by the following claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021-156014 | Sep 2021 | JP | national |
