Patent Application
20240422289
Embodiments of the present disclosure generally relate to a video conferencing systems and related video conferencing methods.
Typically, video-conferencing systems are suitable to allow one or groups of participants to visually conference in different spaces located at different geographical locations. In recent years, video-conferencing has been increasing in popularity due to improvements in high-speed internet, declining costs of video conferencing equipment, and a global need for remote collaboration. As the popularity of video-conferencing has increased, so has the demand for sophisticated video-conferencing systems. Users are now expecting sophisticated video-conferencing systems that are costly, and therefore, found in designated conference areas to be cheaper, available, flexible, and easily installed in any environment used for video conferencing.
Video-conferencing systems have an auto-framing feature that is generally used to detect activity in a video-conferencing environment and adjust the boundaries of the field-of-view (FOV) displayed to remote participants. For example, a video-conferencing system may be able to detect an initial number of users and ensure all the users are clearly in the frame displayed to remote users. Then as the location and/or quantity of users change, the video conferencing system can adjust the FOV of a camera.
Unfortunately, current auto-cropping technologies, used to detect and single out individuals positioned within a FOV of a video conferencing system camera (e.g., auto-framed FOV) for presentation within a video conference, require strict rules or definitions to detect and single out an individual within the camera's FOV. Moreover, current auto-cropping technologies are commonly performed in a sequential manner, which reduces the speed with which the auto-cropping process is performed and creates competing auto-cropping rules or decision factors. For example, first the video-conferencing device may change the FOV of the camera to capture someone outside of the frame first, and then change the FOV to center the users in the FOV. However, fixing an initial problem (i.e., a person is outside of the frame) may lead to another problem. For example, there is no guarantee that in the second step when the users are centered in the FOV will not cause a user to be partially (or fully outside of the frame). Stated differently as each issue in the auto-cropping process is sequentially fixed, a new problem may arise which creates a new iteration of adjustments.
Accordingly, there is a need in the art for methods and a system that allow video-conferencing to auto-crop based on needs indicated by a user, and that solves the other problems highlighted above.
Embodiments of the disclosure include a computer implemented method, comprising generating, by a sensor, a video stream that comprises a series of frames that each include a plurality of objects positioned within a conference environment; determining the objects captured within at least one frame of the video stream; assigning one or more croppings to each of the objects in the at least one frame of the video stream, wherein the assigning of the one or more croppings to each of the objects comprises: determining a plurality of combinations of croppings that include at least one of the objects in the at least one frame; and assigning a first cropping configuration to each of the determined croppings, wherein each of the assigned first cropping configurations include at least one object; adjusting each assigned first cropping configuration to determine a preferred cropping configuration for each of the determined croppings based on a cropping function, wherein the cropping function comprises two or more individual cropping loss values and a respective cropping weight; and transmitting the adjusted croppings to an electronic device for display on a display device.
So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only exemplary embodiments and are therefore not to be considered limiting of its scope, may admit to other equally effective embodiments.
To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation.
Embodiments herein generally relate to video-conferencing systems, and more particularly a method and system for optimally grouping and auto-cropping one or more portions of a field of view (FOV) of a video-conferencing environment based on predefined preferences, such as preferences determined by a user. Generally, embodiments of the disclosure provided herein include a video-conferencing system that is configured to determine optimal croppings of participants within a video conference so that the croppings can be used in the presentation of the participants within a video conference. The video-conferencing system disclosed herein is configured to update the presentation of participants within a video conference so that the optimal presentation of the participants is determine at any given moment in time based on predetermined preferences, while mitigating defects and related unwanted distractions typically generated in the video conference presentation due to the changes provided in the updates. Embodiments of the disclosure allow the predetermined preferences to each be given an importance level rating so as to allow the presentation of the participants to be optimally provided without making unnecessary and/or distracting updates to the video conference presentation, and also help resolve conflicting or competing attributes of related preferences.
The network 124 generally represents any data communications network suitable for the transmission of video and audio data (e.g., the Internet). A communication link 130 is used to support the transmission of video conference feeds that include audio and video streams between the local conference endpoint 101, the network 124, and/or the remote endpoint 122. The communication link 130 may be formed on a network that is connected to a Local Area Network (LAN) present in each video conference location and/or across multiple video conference locations on a Wide Area Network (WAN).
In one embodiment, the communication link 130 utilizes a wired or wireless communication technique to transmit data between the video conferencing locations. Wireless communication techniques can include but are not limited to a cellular phone network, WiFi network, satellite communication network, or other useful wireless communication techniques. Wired communication techniques may include but are not limited to fiber, cable, or DSL type data transmission methods/technologies.
The video conferencing system 105 includes a camera device 106, one or more microphones 107, and a system controller 102. In some embodiments, the video conferencing system 105 also includes a viewfinder device 104 that is used by the system controller 102 to monitor activity in the conference environment 108, e.g., to detect the locations of conference object(s) within the conference environment 108. The viewfinder device 104 may be equipped with a lens and an image sensor to provide an image for processing to the system controller 102. The camera device 106 is used by the system controller 102 to frame a desired field of view (FOV) 111 of camera device 106 based on the detected locations and/or activities of the objects and capture a video stream of the desired view for display at the remote endpoint 122. In some embodiments, the output from the camera device 106 (e.g., video stream) is used by the system controller 102 to monitor activity in the conference environment 108, such as to detect the locations of conference object(s) within the conference environment 108.
In various embodiments, the system controller 102 includes a processor 140, memory 142, input/output (I/O) devices 144, a video streaming device 146, and a communications device 148, which are operably coupled to one another using one or more support circuits (not shown). In some embodiments, a combination of two or more of the processor 140, memory 142, I/O devices 144, video streaming device 146, and the communications device 148 are integrally formed with one another on a single substrate, e.g., to form a system on a chip (SOC) device.
The processor 140 may be any one or combination of a programmable central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an image signal processor (ISP) that is a specialized DSP used for image processing, a programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a neural network coprocessor, or other hardware implementation(s) suitable for performing the methods set forth herein, or portions thereof. The memory 142, coupled to the processor 140, is non-transitory and represents any non-volatile type of memory of a size suitable for storing one or a combination of an operating system 150 and one or more software applications 152.
Examples of suitable memory that may be used as the memory 142 include readily available memory devices, such as random access memory (RAM), flash memory, a hard disk, or a combination of different hardware devices configured to store data. In some embodiments, memory 142 includes memory devices external to the system controller 102 and in communication therewith. In some embodiments, at least one of the one or more software applications 152 are executable by the processor 140 to perform one or more of the methods set forth herein. The one or more software applications may include a cropping software application 156 and a grouping software application 158 that are configured to be executed by the processor 140.
As described above, conventionally, auto-cropping portions of the FOV 111 of the camera device is done sequentially, which is computationally intensive, and often leads to the process of providing a desired presentation of the video conferencing environment from one set of problems to a next set of problems. As discussed above, embodiments of the disclosure provided herein, include a video conferencing system 105 that is configured to determine an optimal presentation of the video conferencing environment by at least adjusting the cropping and grouping of one or more objects or combinations of objects in the conference environment based on predetermined preferences that are stored in memory of the system controller 102. As described herein, the video conferencing system is configured to first execute the cropping software application to determine a combination of croppings, or defined portions of a video frame within a video stream (i.e., sequence of video frames) that includes one or more objects or object groups, to determine a preferred combination of croppings that are to-be used in a presentation of the video conference environment. Each cropping is a region of interest that corresponds to at least a portion of an object that is desired for inclusion into the to-be transmitted video stream that is used in the presentation of the video conferencing environment delivered locally and/or to one or more other video conferencing locations. For example, each cropping is illustrated as a rectangular box surrounding at least a portion of an object. However any desired shape may be used for each cropping and each cropping may be used to surround any desired portion of each object (i.e., participant).
The system controller 102 is configured to determine each potential combination of groupings and croppings for objects or combinations of objects in the conference environment 108. Each combination of croppings includes at least one cropping. Each of the objects are included in at least one cropping of the group of croppings. Each of the croppings includes at least one object. For example, one potential combination of croppings include croppings 120a-120c, as shown in
After determining the preferred cropping, the system controller 102 determines a preferred grouping of the object s in the conferencing environment based on a grouping formula and a maximum quantity of groups. Schematic representations of a method for determining the preferred grouping are illustrated in
As will be discussed further below,
In general, the method 200 is performed using one or more instructions which are executed by use of the processor 140 and resides in the memory 142 of the system controller 102. In some embodiments, a software application running on the system controller is configured to perform each of the activities performed during the completion of method 200.
At activity 202, the method 200 begins by capturing a frame. In one example, the frame is based on a plurality of survey frames. In various embodiments, capturing a frame includes initiating a video conference session by surveying the conference environment 108 by acquiring a plurality of video frames within the generated video stream. These initial video frames are often referred to herein as survey frames. In some embodiments, the video stream is generated using the camera device 106. Typically, the survey frames are analyzed at the beginning of the video-conferencing session to detect objects, such as conference participants 110-114, and periodically throughout the video-conferencing session to detect changes in the video-conferencing session, such as participants leaving, participants changing location, new participants joining, changes in participant activity (changes in who is speaking) and shifting participant engagement levels. Analysis of the survey frames generates survey data which is analyzed by the system controller 102.
At activity 204, the system controller 102 detects each of the objects in the scene captured within the frame (i.e., within the FOV 111). In one example, the objects are participants in the video conference. For example, as shown in
At activity 206, the system controller 102 generates all possible combinations of croppings based on the detected objects. As discussed above, a cropping includes defined portions of a video frame within a video stream (i.e., sequence of video frames) that includes one or more objects. Therefore, based on the number of objects determined in activity 204, the total number of possible combinations of croppings will generally be equal to 2N−1 possibilities, where N is equal to the number of objects. However, the useful and/or desired number of croppings can be less than the total number of possible combinations of croppings, since croppings that are formed between two objects that include an intervening object are generally not useful for determining the optimal croppings due to at least redundancy with other determined croppings. For example, a cropping including the first participant 110 and the third participant 114 must include the second participant 112, which will be covered by a cropping that includes all three participants (e.g., cropping 102d in
At activity 208, the system controller 102 determines preferred croppings for each combination of croppings based on a cropping function. In various embodiments, the attributes of each of the croppings is adjusted until a preferred set of attributes (e.g., size and position) for each cropping is determined. Each combination of croppings is adjusted based on a cropping function that includes a plurality of cropping attributes. As will be described in more detail below, the preferred configuration of a cropping is the combination of cropping attributes that minimize the value of the cropping loss function.
The cropping loss function is used to generate a total combined cropping loss value for each combination of croppings. The total combined cropping loss value for each combination of croppings is determined by determining a set of individual cropping loss values, which are also referred to herein as individual cropping losses, for each possible cropping in each combination of croppings based on a defined set of cropping attributes. For example, a set of individual cropping loss values may be determined for croppings 120a-120f, which are illustrated in
In some embodiments, and for ease of discussion, each individual cropping loss value for each of the cropping attributes is determined in the negative. Stated differently, the higher the cropping loss value the less desirable a defined attribute of a cropping is versus a lower cropping loss value. Moreover, the higher an individual cropping loss value, the worse the cropping is with respect to the corresponding cropping attribute. For example, the more off-center (e.g., asymmetrical) participant(s) are in a cropping, the higher the individual cropping loss value corresponding to asymmetry of objects. In one example, each individual cropping loss value for each cropping attribute may range from a value between 0 and 100 with 0 indicating zero penalty and 100 indicating a maximum penalty. In some embodiments, the range of values of the cropping loss value between the minimum and maximum values is defined by a linear scale. In other embodiments, the maximum penalty that can be achieved is not limited. Examples of how each of these cropping attributes are determined are illustrated in
For example, the head distance 506t, i.e., the distance between the head of first participant 110 and the top of a cropping, in cropping 502b is less than the head distance 506t in cropping 502a. Thus, there is a larger penalty assessed to the individual cropping loss value for cropping 502b due to the reduced head distance 506t. The left distance 506l, i.e., the distance between the left side of first participant 110 and the left side of a cropping, in cropping 502b is less than the left distance 506l in cropping 502a. Thus, there is a larger penalty assessed to the individual cropping loss value for cropping 502b due to the reduced left distance 506l. The right distance 506r, i.e., the distance between the right side of first participant 110 and the right side of a cropping, in cropping 502b is less than the right distance 506r in cropping 502a. Thus, there is a larger penalty assessed to the individual cropping loss value for cropping 502b due to the reduced right distance 506r. The bottom distance 506b, i.e., the distance between the bottom side of first participant 110 and the bottom of a cropping, in cropping 502b is less than the bottom distance 506b in cropping 502a. Thus, there is a larger penalty assessed to the individual cropping loss value for cropping 502b due to the reduced bottom distance 506b. Therefore, the top, bottom, and sides of cropping 502b are adjusted so that they are closer to the respective portion of the participant, resulting in cropping 502b having a higher individual cropping loss value corresponding to a restriction of free space. The adjustment of the top, bottom, and sides of the edge of the cropping each cause the cropping 502b to have a higher individual cropping loss value corresponding to restriction of free space than cropping 502a. However, the degree of penalization for adjusting the edge of the cropping closer to the head is greater than adjusting the edge of the cropping closer to the sides, which is greater than adjusting the edge of the cropping relative to the bottom of the cropping. In one example, the individual cropping loss value for the distance 506t of the cropping 502a may have a value of 10 versus the individual cropping loss value for the distance 506t of the cropping 502b which may have a value of 80, which means that the distance 506t of cropping 502a is preferred over the distance 506t configuration of cropping 502b. In another example, the individual cropping loss value for the distances 506b, 506l and 506r of the cropping 502a may each have a value of 10, 10, 10, respectively, versus the individual cropping loss value for the distances 506b, 506l and 506r of the cropping 502b may have a value of 50, 60, 55, respectively, which means that the distances 506b, 506l and 506r of cropping 502a are preferred over the distances 506b, 506l and 506r configuration of cropping 502b.
Furthermore, if an object is located on the edge of the frame 118, there is no penalty for adjusting the side of the cropping located on the edge. For example, if the first participant 110 is located on the left edge of the frame (e.g., cropping 402b in
Furthermore, an individual cropping loss value corresponding to whether the cropping is larger than the ideal cropping may be determined (not shown). Croppings that are larger than the ideal size may be penalized. The larger a cropping is than the ideal cropping size, the higher the penalty. In some examples, adjusting a cropping to minimize the other individual cropping loss values may cause the cropping to be greater than the ideal cropping size. The attributes of an ideal or preferred cropping size can be defined by attributes stored in memory. The attributes of a cropping that has an ideal or preferred size can be defined by attributes that include, but are not limited to, the area of the cropping (e.g., number of pixels within the edges of the cropping), shape, and aspect ratio of the cropping.
Although five individual cropping attributes are used to determine the cropping loss value for each cropping are discussed above, this is for example purposes only. Each set of individual cropping loss values may include any quantity of individual cropping loss values.
Each of the individual cropping loss values are then inputted into the cropping function to determine an overall cropping loss value for each cropping determined by the system controller 102. In various embodiments, the cropping function includes cropping weights, or cropping coefficients, that correspond to each cropping attribute. In one example, the cropping weights may range in value between 0 and 1. In some examples, a cropping weight may exceed 1. Each individual cropping loss value for each cropping attribute (e.g., FOV restriction, restriction of free space, etc.) is multiplied by its corresponding cropping weight to determine a weighted individual attribute loss value (WIALV), which are then added together to form a weighted individual cropping loss value (WICLV) for each of the croppings. Each of the WICLVs for each combination of croppings are added together generating a total grouping loss value for each combination of croppings.
where C1, C2, C3 and C4 are the cropping weights, and 702a, 702b, 702c and 702d are the individual cropping loss values for each of the cropping attributes. Each cropping weight C1, C2, C3 and C4 correspond to a respective individual cropping loss value 702a, 702b, 702c and 702d. For example, different cropping weights C1, C2, C3 and C4 may be assigned to an asymmetry loss value 702a, FOV restriction loss value 702b, restriction of free space loss value 702c, and hitchhiker presence loss value 702d. In this example, the set of individual cropping attributes determined for croppings 120a, 120b and 120d include four cropping attributes, however, more or less cropping attributes may be used. This process is simultaneously done for all of the other combinations of croppings. The size and shape of each individual cropping is adjusted (i.e., changed) into every possible configuration.
Advantageously each individual cropping is adjusted based on the cropping weights of the cropping formula. This provides guidance for the system controller 102 on how to adjust each of the individual croppings based on user implemented trade-offs between each attribute. For example, if the cropping weight for asymmetry is 0.7 and the cropping weight for the restriction of free space is 0.2, the system controller 102 may prioritize symmetry over the restriction of free space to minimize the WICLVs. Activities 202 through 208 are repeated throughout the video conference.
At activity 209, the software running on the system controller 102 then determines the preferred combination of croppings that include all of the objects within the FOV 111. The preferred combination of croppings will include the one or more croppings formed during activity 208, which when combined together achieves a minimum total combined cropping loss value, such as total combined cropping loss values 761-764, illustrated in
At activity 210, the system controller 102 determines whether a difference between the preferred cropping loss value and also the total combined cropping loss value of the currently used cropping combination (the current cropping loss value) is less than a cropping threshold. Adjusting the attributes of the croppings and/or combination of croppings interferes with the viewing experience of a remote user. Therefore, in some instances it is not worth interrupting the viewing experience for only insignificant improvements. However, for example, if a new participant enters the conference environment 108 the number of objects, the grouping of the objects and/or the frame 118 may need to be adjusted. For example, the cropping threshold may be a value configured to ensure the preferred cropping loss value is at least 50% less (i.e. better) than the current cropping loss value, or at least 40% less (i.e. better) than the current cropping loss value, or at least 30% less (i.e. better) than the current cropping loss value, or at least 20% less (i.e. better) than the current cropping loss value, or at least 10% less (i.e. better) than the current cropping loss value.
If the difference between the preferred cropping loss value and the current cropping loss value is greater than the cropping threshold value, it is not worth interrupting the remote user's viewing experience and the method returns to activity 208. If the difference between a preferred cropping loss value and the current cropping loss value is less than the cropping threshold, the method 200 will proceed to activity 210 and the system controller 102 determines whether the preferred cropping is stable.
At activity 210, the system controller 102 determines whether the duration of time elapsed since the difference between the preferred cropping loss value and the current cropping loss value exceeded a cropping time threshold. For example, as a new participant enters the meeting, the total combined cropping loss value for each combination of croppings will continuously change until the new participant settles into the meeting and takes a seat, causing the preferred cropping to continuously change until the new participant settles in. The system controller 102 is configured not to continuously update the current cropping until the new participant settles in, preventing multiple interruptions to a remote user's viewing experience. Therefore, if the duration of time elapsed does not exceed the cropping time threshold, the method returns to activity 210. If the duration of time elapsed does exceed the cropping time threshold, the method proceeds to activity 214 and the current cropping combination is changed to the preferred cropping.
The WICLVs are constantly changing as frames are continuously surveyed and the croppings are continuously adjusted during a video conference. The greater the improvement of the preferred cropping as the WICLVs are changing, the quicker the cropping should be changed to maximize the viewing experience of a remote user. In various embodiments, to account for this as the difference between the preferred cropping loss value and the cropping threshold increases, the cropping time threshold decreases. Stated differently, the greater the improvement provided by a preferred cropping, the faster it is selected.
After determining all of the possible groupings in activity 209, when the determined number of groupings exceeds the maximum quantity of groupings rule the system controller determines a preferred grouping based on a grouping formula. Schematic representations of a method for determining the preferred grouping are illustrated in
Each combination of groupings includes at least one cropping, and each object or groups of objects are included in at least one cropping that make-up a combination of groupings. For example, referring to
In general, the method 900 is performed using one or more instructions which are executed by use of the processor 140 and resides in the memory 142 of the system controller 102.
At activity 901, the activities performed in method 200 are performed and the preferred combination of croppings are determined in the manner described in relation to
At activity 906, the system controller 102 determines all of the possible combinations of groupings based on a maximum quantity of groupings rule. Stated differently the system controller 102 determines every combination of groupings including a quantity of croppings less than or equal to the maximum quantity of groupings. For example if the maximum quantity of groupings is two, each combination of groupings includes two or less croppings.
Each object or groups of objects are included in at least one cropping of each combination of groupings. Each of the croppings includes at least one object. Stated differently, each participant is included in at least one cropping in each combination of groupings.
At activity 908, the system controller 102 determines preferred croppings for each combination of groupings based on a grouping function. The attributes of each of the croppings is adjusted until a preferred set of attributes for each cropping is determined. Each combination of groupings is adjusted based on a grouping loss function that includes a plurality of grouping attributes. As will be described in more detail below, the preferred configuration of a grouping is the combination of grouping attributes that minimize the value of the grouping function.
The grouping function is used to generate a total combined grouping loss value for each combination of groupings. The total combined grouping loss value is determined by determining a set of individual grouping loss values for each combination of groupings. A set of individual grouping loss values are determined for each cropping in each combination of groupings. For example, a set of individual grouping loss values may be determined for croppings 120a-120f, which are illustrated in
In some embodiments, and for ease of discussion, each individual grouping loss value for each of the grouping attributes is determined in the negative. Stated differently, the higher the grouping loss value the less desirable a defined attribute of a grouping is versus a lower grouping loss value. Moreover, the higher an individual grouping loss value, the worse the grouping is with respect to the corresponding grouping attribute. For example, the greater the horizontal distance between participants are in a cropping, the higher the individual grouping loss value. In one example, each individual grouping loss value for each grouping attribute may range from a value between 0 and 100 with 0 indicating zero penalty and 100 indicating a maximum penalty. In some embodiments, the range of values of the grouping loss value between the minimum and maximum values is defined by a linear scale. Examples of how each of these grouping attributes are determined are illustrated in
Each of the individual grouping loss values are then inputted into the grouping function to determine an overall grouping loss value for each cropping in each combination of groupings determined by the system controller 102. In various embodiments, the grouping function includes grouping weights, or grouping coefficients, that correspond to each grouping attribute. In one example, the grouping weights may range in value between 0 and 1. In some examples, a grouping weight may exceed 1. Each individual grouping loss value for each grouping attribute (e.g., distance between objects, space restriction, vertical shift between objects, etc.) is multiplied by its corresponding grouping weight to determine an individual weighted grouping attribute loss values (IWGALVs), which are then added together to form individual weighted grouping loss values (IWGLVs) for each of the croppings. Each of the IWGLVs for each combination of groupings are added together generating a total grouping loss value for each combination of groupings.
where G1, G2, G3 and G4 are the grouping weights, and 802a, 802b, 802c and 802d are the individual grouping loss values for each of the grouping attributes. Each grouping weight G1, G2, G3 and G4 correspond to a respective individual grouping loss value 802a, 802b, 802c and 802d. For example, different grouping weights G1, G2, G3 and G4 may be assigned to a distances between objects grouping attribute 802a, a space restriction grouping attribute 802b, a vertical shift grouping attribute 802c, and a horizontal distance between objects grouping attribute 802d, respectively.
Although the quantity of grouping attributes determined is four, this is for example purposes only, any quantity of grouping attributes could be used. The quantity of grouping attributes can be greater than or less than four. The same four grouping attributes may be evaluated for each of the possible groupings determined by the software running on the system controller 102. This process is simultaneously done for all of the other combinations of groupings. The size and shape of each individual cropping is adjusted (i.e., changed) into every possible configuration based on the maximum grouping rule.
Advantageously, each individual grouping is adjusted based on the grouping weights of the grouping formula provided by a user. Advantageously, this provides guidance for the system controller 102 on how to adjust each of the individual groupings. For example if the grouping weight (G1) for a vertical shift is assigned a weight of 0.7 and the grouping weight (G2) for horizontal distance between groupings loss is assigned a weight of 0.2, the system controller 102 will tend to prioritize vertical shift over the horizontal distance between groupings since the weighting tends to minimize the total combined grouping loss values.
At activity 910, the software running on the system controller 102 then determines the preferred combination of groupings that include all of the objects within the FOV 111. The preferred combination of groupings will include the one or more croppings formed during activity 908, which when combined together achieves a minimum total combined grouping loss value, such as total combined grouping loss values 861-864, illustrated in
At activity 912, the system controller 102 determines whether a difference between the preferred grouping loss value and also the total combined grouping loss value of the currently used grouping combination (the current grouping loss value) is less than a grouping threshold. Adjusting the attributes of the croppings and/or combination of groupings interferes with the viewing experience of a remote user. Therefore, in some instances it is not worth interrupting the viewing experience for only insignificant improvements. However, for example, if a new participant enters the conference environment 108 the number of objects, the grouping of the objects and/or the frame 118 may need to be adjusted. For example, the grouping threshold may be a value configured to ensure the preferred grouping loss value is at least 50% less (i.e. better) than the current cropping loss value, or at least 40% less (i.e. better) than the current cropping loss value, or at least 30% less (i.e. better) than the current cropping loss value, or at least 20% less (i.e. better) than the current cropping loss value, or at least 10% less (i.e. better) than the current cropping loss value.
If the difference between the preferred grouping loss value and the current grouping loss value is greater than the grouping threshold value, it is not worth interrupting the remote user's viewing experience and the method returns to activity 908. If the difference between a preferred grouping loss value and the current grouping loss value is less than the grouping threshold, the method 900 will proceed to activity 914 and the system controller 102 determines whether the preferred grouping is stable.
At activity 914, the system controller 102 determines whether the duration of time elapsed since the difference between the preferred grouping loss value and the current grouping loss value exceeded a grouping time threshold. For example, as a new participant enters the meeting, the total combined grouping loss value for each combination of grouping will continuously change until the new participant settles into the meeting and takes a seat, causing the preferred grouping to continuously change until the new participant settles in. The system controller 102 is configured not to continuously update the current grouping until the new participant settles in, preventing multiple interruptions to a remote user's viewing experience. Therefore, if the duration of time elapsed does not exceed the cropping time threshold the method returns to activity 912. If the duration of time elapsed does exceed the grouping time threshold, the method proceeds to activity 916 and the current grouping combination is changed to the preferred grouping.
The preferred grouping loss value is constantly changing as frames are continuously surveyed and the grouping are continuously adjusted during a video conference. The greater the improvement of the preferred grouping, the quicker the grouping should be changed to maximize the viewing experience of a remote user. In various embodiments, to account for this as the difference between the preferred grouping loss value and the grouping threshold increases, the grouping time threshold decreases. Stated differently, the greater the improvement provided by a preferred grouping the faster it is selected.
As described above, the maximum quantity of groupings can be indicated by a user. In some examples, the system controller 102 may be configured to form a maximum number of groupings that is greater than the maximum quantity of groupings indicated by the user. In this case, the system controller 102, based the grouping function, may determine that the viewing experience would improve if the user increased the indicated maximum quantity of groupings. In response, the system controller 102 may recommend the improved maximum quantity of groupings to the user in any suitable method such as a pop-up window. For example, if the user indicated a maximum quantity of groupings is 2, but the system controller 102 determines that a combination of groupings including 3 grouping has a lower total combined grouping loss value then the preferred grouping, the system controller 102 may recommend that the user increase the maximum quantity of groupings.
While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
This patent application claims the benefit of U.S. Provisional Patent Application No. 63/472,800, filed Jun. 13, 2023, which is incorporated by reference herein.
| Number | Date | Country | |
|---|---|---|---|
| 63472800 | Jun 2023 | US |
