Patent Application
20210352244
The present application relates to technically inventive, non-routine solutions that are necessarily rooted in computer technology and that produce concrete technical improvements.
As COVID-19 has ground much of the world to a standstill, the number of people using video chat for work and play has skyrocketed. One of the resulting phenomena is large groups of people getting together for a “virtual happy hour” or a “virtual birthday party” or similar. As understood herein, a problem is that video chat applications are not designed to make communication between that many people actually work. Every person's video is visible at the same size (often very small), everyone can hear everyone else talking at the same volume, everything said and done is visible and audible to everyone else at all times.
The result of the above is that large group video chats end up quickly disintegrating into many separate smaller video chats. Subgroups spin up their own calls with select people. There is no easy way for others to know those calls exist, or who is in them. There is no way to easily move between calls. There is no easy way to seamlessly rejoin the main call.
In contrast, in a real-life large group meeting there may be dozens if not hundreds of people in a room, but an attendee would not be talking to all of them at once. People tend to gravitate to smaller groups of a handful of people, having a relatively constrained conversation. People would not be able to see or hear most of the people in the party, even if they are technically “there.” Other groups would be scattered around the space. If a person wishes to move from one conversation to another, the person walks over to a different group, which is possible because people can see who is talking to whom from across the room and thus can preemptively decide whether to join them without first hearing what they're talking about. Moreover, people can engage in quick private conversations by drawing close to each other and whispering or stepping away from the group for a minute to talk, then quickly re-join. People can stroll around the room to check out who is there, lurking nearby to sample the conversation before deciding to join it. If one group is not talking about something interesting, a person could stroll past another. Real life social dynamics such as these are sought to be provided in a virtual chat.
Accordingly, an assembly includes at least one display, at least one network interface, and at least one processor configured with instructions to receive information via the network interface pertaining to plural chat participants. The instructions are executable to, using the information, present on the display a user interface (UI). The UI includes a map window showing icons or avatars each respectively representing a chat participant, such that all of the plural chat participants are represented on the map window by a respective icon or avatar. The UI further includes a video chat window that is separate from the map window and that presents videos of respective users in a subgroup of chat participants. The users in the subgroup are less than all of the plural chat participants, and the subgroup may be established based at least in part on proximity. In example implementations, the proximity is based at least in part on proximity of respective icons or avatars to each other in the map window.
In some embodiments the assembly includes at least one speaker and the instructions are executable to present on the speaker audio from users in the subgroup but not from other chat participants not in the subgroup.
In example embodiments the UI can includes a list window presenting a list of all chat participants represented in the map window.
In non-limiting implementations the instructions may be executable to present the map window in a primary region of the display and present the video chat window in a sidebar portion of the display, and then responsive to user input, present the map window in the sidebar portion of the display and present the video chat window in the primary region of the display.
In examples discussed further herein, the display may include a two-dimensional video display, or it may include a virtual reality (VR) or augmented reality (AR) head-mounted display (HMD).
In some implementations the instructions can be executable to configure the video chat window of the respective users in the subgroup of chat participants in a public mode, in which a first chat participant moving a respective icon into proximity of the subgroup can see videos of the users in the subgroup and hear audio from the users in the subgroup. The instructions may be executable to configure the video chat window of the respective users in the subgroup of chat participants in a private mode, in which the first chat participant moving a respective icon into proximity of the subgroup cannot see videos of the users in the subgroup or hear audio from the users in the subgroup. The instructions also may be executable to, responsive to a request from the first chat participant to enter the subgroup while in the private mode, present on a display of at least one of the users in the subgroup an interface to reject or accept the first chat participant.
In examples, the instructions may be executable to present on a display associated with a first chat participant a list of chat participants represented by respective avatars or icons in the map window. The instructions also may be executable to move the respective avatar or icon of the first chat participant to a location in the map window of an avatar or icon of a second chat participant selected from the list.
In another aspect, a method includes presenting on respective display devices of respective chat participants a video chat application simulating real-life social dynamics at least in part by providing an onscreen map to each display device showing pawns representing respective chat participants. The method includes permitting users in subgroups of the chat participants to engage in conversations while viewing videos and hearing audio from members of the respective subgroups. Further, the method includes moving chat participants between subgroups responsive to the respective chat participants moving their pawns on the map.
In another aspect, a system includes at least one video chat server and plural devices communicating with the chat server. Each device is associated with a respective user. The system also includes at least one processor configured with instructions to present on each device a map with pawns representing the respective users. The instructions are executable to present on at least one device a video chat window along with the map and showing video of at least first and second users based on respective first and second pawns being proximate to each other on the map.
The details of the present disclosure, both as to its structure and operation, can be best understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which:
This disclosure relates generally to computer ecosystems including aspects of consumer electronics (CE) device-based user information in computer ecosystems. A system herein may include server and client components, connected over a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices including portable televisions (e.g. smart TVs, Internet-enabled TVs), portable computers such as laptops and tablet computers, and other mobile devices including smart phones and additional examples discussed below. These client devices may operate with a variety of operating environments. For example, some of the client computers may employ, as examples, operating systems from Microsoft or Unix or Apple, Inc. or Google. These operating environments may be used to execute one or more browsing programs, such as a browser made by Microsoft or Google or Mozilla or other browser program that can access web applications hosted by the Internet servers discussed below.
Servers may include one or more processors executing instructions that configure the servers to receive and transmit data over a network such as the Internet. Or, a client and server can be connected over a local intranet or a virtual private network. A server or controller may be instantiated by a game console such as a Sony PlayStation®, a personal computer, etc.
Information may be exchanged over a network between the clients and servers. To this end and for security, servers and/or clients can include firewalls, load balancers, temporary storages, and proxies, and other network infrastructure for reliability and security. One or more servers may form an apparatus that implement methods of providing a secure community such as an online social website to network members.
As used herein, instructions refer to computer-implemented steps for processing information in the system. Instructions can be implemented in software, firmware or hardware and include any type of programmed step undertaken by components of the system.
A processor may be a single- or multi-chip processor that can execute logic by means of various lines such as address lines, data lines, and control lines and registers and shift registers.
Software modules described by way of the flow charts and user interfaces herein can include various sub-routines, procedures, etc. Without limiting the disclosure, logic stated to be executed by a particular module can be redistributed to other software modules and/or combined together in a single module and/or made available in a shareable library.
Present principles described herein can be implemented as hardware, software, firmware, or combinations thereof; hence, illustrative components, blocks, modules, circuits, and steps are set forth in terms of their functionality.
Further to what has been alluded to above, logical blocks, modules, and circuits described below can be implemented or performed with a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic device such as an application specific integrated circuit (ASIC), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor can be implemented by a controller or state machine or a combination of computing devices.
The functions and methods described below, when implemented in software, can be written in an appropriate language such as but not limited to Java C# or C++, and can be stored on or transmitted through a computer-readable storage medium such as a random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage such as digital versatile disc (DVD), magnetic disk storage or other magnetic storage devices including removable thumb drives, etc. A connection may establish a computer-readable medium. Such connections can include, as examples, hard-wired cables including fiber optics and coaxial wires and digital subscriber line (DSL) and twisted pair wires.
Components included in one embodiment can be used in other embodiments in any appropriate combination. For example, any of the various components described herein and/or depicted in the Figures may be combined, interchanged or excluded from other embodiments.
“A system having at least one of A, B, and C” (likewise “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.
Now specifically referring to
Accordingly, to undertake such principles the AVDD 12 can be established by some or all of the components shown in
In addition to the foregoing, the AVDD 12 may also include one or more input ports 26 such as, e.g., a USB port to physically connect (e.g. using a wired connection) to another CE device and/or a headphone port to connect headphones to the AVDD 12 for presentation of audio from the AVDD 12 to a consumer through the headphones. The AVDD 12 may further include one or more computer memories 28 that are not transitory signals, such as disk-based or solid-state storage (including but not limited to flash memory). Also, in some embodiments, the AVDD 12 can include a position or location receiver such as but not limited to a cellphone receiver, GPS receiver and/or altimeter 30 that is configured to e.g. receive geographic position information from at least one satellite or cellphone tower and provide the information to the processor 24 and/or determine an altitude at which the AVDD 12 is disposed in conjunction with the processor 24. However, it is to be understood that that another suitable position receiver other than a cellphone receiver, GPS receiver and/or altimeter may be used in accordance with present principles to e.g. determine the location of the AVDD 12 in all three dimensions.
Continuing the description of the AVDD 12, in some embodiments the AVDD 12 may include one or more cameras 32 that may be, e.g., a thermal imaging camera, a digital camera such as a webcam, and/or a camera integrated into the AVDD 12 and controllable by the processor 24 to gather pictures/images and/or video in accordance with present principles. Also included on the AVDD 12 may be a Bluetooth transceiver 34 and other Near Field Communication (NFC) element 36 for communication with other devices using Bluetooth and/or NFC technology, respectively. An example NFC element can be a radio frequency identification (RFID) element.
Further still, the AVDD 12 may include one or more auxiliary sensors 37 (e.g., a motion sensor such as an accelerometer, gyroscope, cyclometer, or a magnetic sensor, an infrared (IR) sensor, an optical sensor, a speed and/or cadence sensor, a gesture sensor (e.g. for sensing gesture command, etc.) providing input to the processor 24. The AVDD 12 may include still other sensors such as e.g. one or more climate sensors 38 (e.g. barometers, humidity sensors, wind sensors, light sensors, temperature sensors, etc.) and/or one or more biometric sensors 40 providing input to the processor 24. In addition to the foregoing, it is noted that the AVDD 12 may also include an infrared (IR) transmitter and/or IR receiver and/or IR transceiver 42 such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVDD 12.
Still referring to
The example non-limiting first CE device 44 may be established by any one of the above-mentioned devices, for example, a portable wireless laptop computer or tablet computer or notebook computer or mobile telephone, and accordingly may have one or more of the components described below. The second CE device 46 without limitation may be established by a wireless telephone. The second CE device 46 may implement a portable hand-held remote control (RC). The second CE device 46 may implement virtual reality (VR) and/or augmented reality (AR) a head-mounted display (HMD). The CE devices 44, 46 may include some or all of the components illustrated in the case of the AVDD 12.
At least one server 50 may include at least one server processor 52, at least one computer memory 54 such as disk-based or solid-state storage, and at least one network interface 56 that, under control of the server processor 52, allows for communication with the other devices of
Accordingly, in some embodiments the server 50 may be an Internet server and may include and perform “cloud” functions such that the devices of the system 10 may access a “cloud” environment via the server 50 in example embodiments. Or, the server 50 may be implemented by a game console or other computer in the same room as the other devices shown in
Devices discussed herein may include some or all, as appropriate, of the various components shown in
Turn now to
A pawn's location on the map relative to other pawns determines which video chats will be visible and which audio will be audible to any given user. In other words, proximity to others determines who users see and hear, just like in real life. People who are farther away on the map are still “there,” but audio and video of the respective users are not presented to a particular user unless the particular user is “close.”
To illustrate,
Because Andy, Bob, and Charlie are near each other on the map, they can all see each other's video chat 110 and hear each other's audio. But Dan, Edward, and Frank are too far away on the other corner of the map, so their video and audio is hidden and muted. In other words, Andy, Bob, and Charlie can see and hear each other, but they cannot see or hear Dan, Edward, and Frank except for the pawns 102 in the map 100 shown in the sidebar 108.
Likewise, Dan, Edward, and Frank are near each other, so they can see and hear each other, but cannot see or hear Andy, Bob, and Charlie.
Now, suppose Andy is tiring of his conversation with Bob and Charlie. He can look at the map and see that Dan, Edward, and Frank are talking together somewhere else. He is friends with those guys, so he knows he can feel comfortable joining their conversation. Because he can see on the map who is talking together ahead of time, he can preemptively decide to join before actually joining.
To leave his conversation with Bob and Charlie, Andy simply needs to navigate far enough away from them on the map. As his distance from them grows, their video chat windows 110 shrink as shown in
Once Andy has exceeded a maximum distance, Bob and Charlie's video and audio is hidden altogether on Andy's device. Andy is now distanced from the other users as shown in
Meanwhile, as illustrated in
Refer now to
The whisper is just for a quick chat, a few words here and there. But what if Andy and Dan want to have a more in-depth private conversation? Continually using the whisper is inconvenient and potentially disruptive to the larger group conversation. Andy and Dan can choose to simply “step aside” to have a private conversation. To do this, they both simply move their pawn on the map to an area further away from any other group, as shown in
Any conversation between two people alone can be private by default. What that means is, if another pawn moves close to a pair of pawns on the map, that third pawn will not be able to see or hear the video of the pair right away. For example, Andy and Dan step away to have a private chat. Frank moves his pawn near them. Frank will not be able to see or hear Andy and Dan by default.
For example, if Andy and Dan are in a private conversation, Andy can click Options and select Make Public. Dan receives the request. If he accepts, the conversation is public. Once public, any other pawn can move close on the map and be automatically joined to the conversation. If Dan rejects the request, the conversation remains private.
Once made public, a conversation can be made private again in the same way. One member of the pair selects Options and chooses Make Private. Once the other accepts that Make Private request, the conversation is private again.
Returning to Frank, who has approached the private chat between Andy and Dan, Frank may be permitted to request to join the conversation using one or more request modes, with examples referred to herein as “wave” and “knock”.
If Andy chooses to reject Frank's Wave, then Frank receives back a message 126 as shown in
Present principles understand that a user (such as Frank in the example below) may wish to urgently join a private conversation, and so a Knock function is provided. As with a Wave, Frank can select Andy or Dan's pawn and select Knock from the menu 120 shown in
The selected user can then allow or reject that Knock. Just like with a wave, Andy accepting Frank's Knock allows Frank to see/hear Andy and Dan, and vice versa. If Andy was to reject Frank's Knock, then Frank would receive a message back saying he cannot join.
A Knock can also be accepted in more than one way. When Andy accepts Frank's Knock, he can choose to do it for a limited time, for example one minute, five minutes, ten minutes, or unlimited. This gives Andy and Dan the option to include Frank in their private conversation for a short time to hear what he has to say, before the conversation reverts back to being private.
At any time, Andy can select the timer to either add more time or make it unlimited.
Refer now to
In the example shown, a shout selector 140 is presented on the people menu or list 104. The user selecting shout appear in every group's video chat, regardless of privacy. However, it is unidirectional in that everyone can hear and see video and audio of the person shouting, but the person shouting cannot see or hear everyone else. This preserves the privacy of the private groups.
For example, say Andy, Dan, and Frank are in a conversation, and Andy wants to let everyone in the room know he is leaving. He can use a Shout to say his goodbyes. This would make Andy's video appear in Bob and Charlie's chat, even though Andy is nowhere near them. Andy can say his goodbyes to everyone, then log off for the night.
Shouts can be muted or ignored. If Andy uses a shout, Bob and/or Charlie can mute him, which kills his audio while leaving his video still visible. Or they can ignore him, which will kill the audio and video. Mute and ignore are both reversible, to bring back the Shout's audio and/or video on demand.
Refer now to
To do that, Andy can look at the list 104 of all the participants in the party, for example in a sidebar that lists the participants alphabetically. Andy can simply find Bob in the list as shown at 150 in
From there, all of the above-discussed proximity and privacy rules apply—if Bob is in a private conversation, Andy will still need to Wave or Knock before he can join.
Say Andy actually does find Bob on the map. In other words, he does not have to resort to using the list view. Then Andy can select Bob's pawn and choose Wave or Knock. If Bob accepts the Wave/Knock, Andy will automatically be teleported to Bob's location on the map and will be joined into the conversation.
In the example shown, assume Andy, Dan, and Frank are in a conversation. They start discussing something that Bob knows a lot about. Andy can select Bob from the list 104 in the sidebar and select Pull 156.
In response,
If Bob declines the pull solicitation,
As mentioned above, if two people are in a private conversation and a third person requests to join, a Wave or Knock may be accepted by the conversationalists or declined. In
In other words, once one member of a conversation requests to make it private, the others in the conversation have the option to either participate in the private conversation or leave the conversation and go find someone else to talk to.
Members who reject the privacy request may be asked to confirm if they want to leave the conversation. If they confirm, they will be unable to see/hear those who accepted the privacy request as shown in
The reverse mechanic (private-to-public) is similar. Refer to
Note that face capture may be executed at block 1400 as well, mapped onto a 3D skeleton image, and provided to the devices of other chat participants as the video of the wearer of the HMD.
Note further that assets in the map may define screen space. For example, if the map models a dinner party, a participant's proximity to others around the table defines to whom the participant may speak and hear.
It is to be understood that the map need not emulate one big open area and instead can alternatively emulate public and private rooms. A room may be used to apply more specific, stringent rules about who can participate, and how. For example, in a movie room, a feature film may be playing, with member audio disabled by default so people watching the movie are not disturbed by people talking. However, the whisper function still works, so people can have short private conversations, just like in a real movie theater.
In contrast, a game room may present a sports game or a video game in which member audio is not disabled to allow members to cheer and chant and otherwise express their enthusiasm for what they are watching. If a video game is being played, then control of that game can be passed to another member of the room via Share Play. This can allow for things like a virtual couch or virtual arcade experience, with players trading off control of the game while others watch and cheer.
A music room is contemplated in which one user is in control of the music that is playing, emulating a DJ. Anyone who enters can hear the music in addition to their voice chat. The Options menu can include options like Ask to DJ, Request a Song, etc.
A panel discussion room may be instantiated in which a few members (the “panel”) are allowed to talk and can be seen. Anyone who enters the room is part of the audience. The audience can see and hear the panel, but their audio and video are disabled by default. The Options menu might include an Option like Raise Hand or Ask Question to send a request to the panel moderator. If accepted, the requestor's mic and video are shown to the panel and the audience for the duration of the question.
A private meeting room can be associated with a whitelist of people who are allowed to enter. It may also conceal the identity of the people in the room, by anonymizing the pawns. Anyone on the whitelist can enter the room. The existence of the room may also be gated by the whitelist—if a member is not on the list, the member does not know if the room's existence.
A presentation room can have a presenter and an audience. The presenter may be the only one whose video and audio is visible. In addition, the presenter can share his screen and everyone in the audience can see it. Like panel discussions, this can include a Raise Hand or Ask Question option.
Restricted rooms can have rules about which members can enter, such as age restrictions (adults-only rooms). They may also have a warning that the user must accept before joining, so warn of potentially offensive topics or situations such as nudity or adult language.
Without limitation, present principles may be applied to virtual house parties, in which participants are emulated as being located in a big empty map, free to move about and socialize. Present principles may apply to a virtual convention with thousands of attendees, with the map being a replica of a convention center floor plan. Chat participants navigate around the show floor, visit different booths, listen to demos or panels, network with colleagues, have private meetings, etc.
Or, the map may be skinned to match the layout of an office floor plan, with cubicles, meeting rooms, common areas, etc. Chat participants can have virtual stand-ups, private meetings, public lunches, social events, happy hours, etc.
A further application relates to speed dating or speed networking in which the map is configured like a speed dating event, with different tables set up around the room. Chat participants on one side of the table could say static. Every few minutes pawns on the other side of the table can be rotated to move them one table to the left or right. Each would be a private pair conversation where people can get to know each other. They can exchange contact info if they want to keep the conversation going.
While particular techniques are herein shown and described in detail, it is to be understood that the subject matter which is encompassed by the present application is limited only by the claims.
