Patent Grant
11800213
Embodiments of the present disclosure generally relate to video conferencing systems.
Video conferencing has become more popular in recent years, thanks in large part to the declining costs of electronic equipment associated therewith and the proliferation of high-speed Internet. Often, individuals or organizations who regularly use video conferencing will designate a specific conference room (or other physical space) to include technological resources dedicated solely to the task of video conferencing, thus providing a designated video conferencing environment.
Generally, designated video conferencing environments will include at least one camera positioned to capture a video stream of all or a portion of the designated conferencing environment and thus all or some of the video conference participants situated therein. More sophisticated video conferencing systems may be configured to detect activity in a video conferencing environment and adjust the boundaries of an image (field-of-view) displayed to a remote conference participant accordingly. For example, some video conferencing systems may be configured to detect an initial number of user participants within a larger physical space and automatically reframe the captured video stream (e.g., optically zoom in) around those participants to improve the viewing experience of remote conference participants. Typically, once a camera's field-of-view has been reframed about the initial conference participants using optical zoom, the camera is no longer useful for monitoring areas outside the of the framed space for changes in information. Thus, systems that are even more sophisticated may use a second camera to monitor the unframed spaces in the conference room for changes in information, such as new participants entering the room outside of the zoomed in field-of-view of the first camera. This new information may be used in directing the first camera to subsequently reframe (e.g., optically zoom out) to capture video stream of a larger area which includes those newly entered conference participants. Unfortunately, the cost, time, and technical complexity of such a two-camera system may be prohibitive to some potential users who otherwise desire the improved user experience of auto-framing and video conferencing system technology.
Accordingly, there is a need in the art for an advanced single camera video conferencing system, and video conferencing methods related thereto, that solves the problems described above.
The present disclosure generally provides for advanced single camera video conferencing systems, and methods related thereto. The advanced single camera video conferencing systems typically feature a hybrid optical/digital camera, herein a camera device, having a controller that is configured to execute one or more of the methods set forth below.
In one embodiment, a video conferencing method using an advanced single camera video conferencing system is provided. The method includes using a camera device to define an actual field-of-view where the actual field-of-view optically frames a first portion of a video conferencing environment. The method further includes using the same camera device to digitally frame and stream a second portion of the video conferencing environment to provide an apparent field-of-view. Typically the same image sensor of the same camera device is used to frame the actual field-of-view and the apparent field-of-view. The apparent field-of-view is generally acquired from only a portion of the actual field-of-view of the image sensor. Thus, the apparent field-of-view will be encompassed within the actual field-of-view and an excluded region will be encompassed within a portion of the actual field-of-view which is outside of the boundaries of the apparent field-of-view. The video conferencing method further includes generating a video stream of the apparent field-of-view. In some embodiments, generating a video stream of the apparent field-of-view includes acquiring video data from the portion of the actual field-of-view of the image sensor that defines the boundaries of the apparent field-of-view and not acquiring video data from the excluded region. Acquiring video data from only the portion of the actual field-of-view of the image sensor that defines the boundaries of the apparent field-of-view beneficially facilitates seamless integration of the method with readily available video conferencing software applications, such as Microsoft® Skype®, Apple® FaceTime® and applications available from Zoom® Video Communications, Inc. Here, the video conferencing method further includes generating a plurality of survey frames of the actual field-of-view and analyzing the plurality of survey frames to generate survey data. The survey data may be used to detect changes within the first portion of the video conferencing environment over time.
Embodiments of the present disclosure may further provide a method of monitoring a video conferencing environment using an advanced single camera video conferencing system. The method includes framing a first portion of a video conferencing environment to provide an actual of view, generating a video stream of an apparent field-of-view encompassed within the actual field-of-view, and transmitting the video stream of the apparent field-of-view to a video conferencing software application which is being executed on a user device. The method further includes periodically generating one or more frames of the actual field-of-view, extracting the one or more frames of the actual field-of-view from the video stream before the video stream is transmitted to the video conferencing application, and analyzing the one or more periodically generated frames of the actual field-of-view to generate survey data. Here, the video stream includes one or more pluralities of sequentially generated frames of the apparent field-of-view. The respective frames of the actual field-of-view and video stream of the apparent field-of-view are both generated using the same camera device. In some embodiments, the respective frames of the actual field-of-view and the video stream of the apparent field-of-view are both generated using the same image sensor of the same camera device.
Embodiments of the present disclosure may further provide a method of framing a portion of video conferencing environment to provide a desired apparent field-of-view while maintaining a desired threshold image resolution of a video stream. The method allows for video streaming of a desired apparent field-of-view and concurrent monitoring of the video conferencing environment in portions excluded from the desired apparent field-of-view using an advanced single camera video conferencing system. The method includes optically framing a portion of a video conferencing environment to provide an actual field-of-view, determining a desired threshold image resolution of a to-be-transmitted video stream of a desired apparent field-of-view, digitally zooming up to a desired threshold image resolution to provide a digitally zoomed image, and comparing the digitally zoomed image to the desired apparent field-of-view to determine a difference therebetween. In some embodiments, the method further includes optically zooming the camera device based on the difference between the digitally zoomed image and the desired apparent field-of-view to provide the desired apparent field-of-view at the desired resolution. In some embodiments, the desired threshold image resolution is derived from a dynamically changing bandwidth that is available for transmitting the video stream. Typically the same image sensor of the same camera device is used to frame both the actual field-of-view and the apparent field-of-view where the apparent field-of-view is acquired from only a portion of the actual field-of-view detected by the image sensor. Thus, the apparent field-of-view will be encompassed within the actual field-of-view and an excluded region will be encompassed within a portion of the actual field-of-view which is outside of the boundaries of the apparent field-of-view.
Embodiments of the present disclosure may further provide a video conferencing method, comprising defining, by use of a camera device, a first actual field-of-view, wherein the first actual field-of-view optically frames a first portion of a video conferencing environment, digitally framing, by use of the camera device, a second portion of the video conferencing environment to provide a first apparent field-of-view, wherein the first apparent field-of-view is encompassed within the first actual field-of-view, generating, by use of the camera device, a video stream of the first apparent field-of-view, generating, by use of the camera device, survey data comprising a plurality of survey frames that each comprises at least a portion of the first actual field-of-view that is different from the first apparent field-of-view, and detecting changes in the survey data over time. The process of digitally framing the second portion of the video conferencing environment to provide the first apparent field-of-view may include determining a desired threshold image resolution of a to-be-transmitted video stream of the first apparent field-of-view, digitally zooming, using the camera device, up to the desired threshold image resolution to provide a digitally zoomed image, comparing the digitally zoomed image and the first apparent field-of-view to determine a difference therebetween, and based on the difference between the digitally zoomed image and the first apparent field-of-view, optically zooming to the first apparent field-of-view. The desired threshold image resolution may be determined by a display resolution of a system that is configured to receive the to-be-transmitted video stream at a remote location or by a dynamically changing bandwidth which may be used to transmit the video stream.
Embodiments of the present disclosure may further provide a method of monitoring a video conferencing environment, comprising optically framing, by use of a camera device, a first portion of a video conferencing environment to provide a first actual field-of-view, generating, by use of the camera device, a video stream of a first apparent field-of-view, wherein the first apparent field-of-view is encompassed within the first actual field-of-view, and the video stream comprises one or more pluralities of sequentially acquired frames of the first apparent field-of-view, transmitting the video stream of the first apparent field-of-view to a user device that includes a video conferencing software application stored in memory, periodically interrupting the video stream of the first apparent field-of-view to generate, using the camera device, one or more survey frames that comprise at least a portion of the first actual field-of-view that is different from the first apparent field-of-view, extracting the one or more survey frames from the video stream before the video stream is transmitted to the user device, and storing the one or more survey frames captured during each of the periodic interruptions as survey data.
Embodiments of the present disclosure may further provide a video conferencing method, comprising optically framing, by use of a camera device, a portion of a video conferencing environment to provide a first actual field-of-view, determining a desired threshold image resolution of a to-be-transmitted video stream of a desired apparent field-of-view, digitally zooming, by use of the camera device, up to a desired threshold image resolution to provide a digitally zoomed image, comparing the digitally zoomed image to the desired apparent field-of-view to determine a difference therebetween, and based on the difference between the digitally zoomed image and the desired apparent field-of-view, optically zooming, by use of the camera device, to the desired apparent field-of-view.
Embodiments of the present disclosure may further provide video conferencing system comprising a camera device and a non-transitory computer readable medium. The camera device comprises an image sensor and a lens, wherein one or both of the lens and the image sensor are movable with respect to the other to change a focal length of the camera device. The non-transitory computer readable medium includes instructions which when executed by a processor are configured to perform a method comprising optically framing a first portion of a video conferencing environment to provide a first actual field-of-view, digitally framing a second portion of the video conferencing environment to provide a first apparent field-of-view, wherein the first apparent field-of-view is encompassed within the first actual field-of-view, and generating a video stream of the first apparent field-of-view. In some embodiments, the camera device comprises a processor for executing the method and the non-transitory computer readable medium having instructions for the method stored thereon. In other embodiments, the processor for executing the method and/or the non-transitory computer readable medium are comprised in a user device in communication with the camera device.
Embodiments of this disclosure may further provide a computer implemented method for preventing undesired and frequent adjustments to the boundaries of a video stream of a physical environment based on relatively minor detected changes therein. The method includes generating a crop-failure score for a current apparent field-of-view, comparing the crop-failure score or an accumulated crop-failure score to a crop-failure tolerance limit, determining boundaries of a desired apparent field-of-view, and altering the video stream when the crop-failure score or the accumulated crop-failure score is equal to or outside of the crop-failure tolerance limit. Here, the crop-failure score is based on a location of one or more participants in a portion of a physical environment relative to the boundaries of the current apparent field-of-view of a camera device. The accumulated crop-failure score generally comprises a plurality of generated crop-failure scores. The boundaries of a desired apparent field-of-view are based on the locations of the one or more participants in the portion of the physical environment. In some embodiments, the portion of the physical environment is defined by an actual field-of-view of a camera device.
Embodiments of this disclosure may further provide a computer implemented method for adjusting the boundaries of a video stream of a physical environment based on detected changes therein. The method facilitates alterations to the video stream to improve the viewing experience of a remote user by reframing an apparent field-of-view of the video stream for significant events while concurrently preventing frequent and undesirable adjustments to the apparent field-of-view that might cause physical discomfort, such as motion related sickness, for the remote viewer. The method includes generating a crop-failure score for a current apparent field-of-view of a video stream generated by a camera device, comparing the generated crop-failure score or an accumulated crop-failure score to a crop-failure tolerance limit, and determining when to adjust the crop of the apparent field-of-view. Generally determining when to adjust the crop of the apparent field-of-view includes determining the boundaries of a desired apparent field-of-view of a video stream when the generated crop-failure score is equal to or outside of the crop-failure tolerance limit. In some embodiments, the method further includes determining a difference between at least one attribute of the desired apparent field-of-view with the current apparent field-of-view and comparing the determined difference between the at least one attribute of the desired apparent field-of-view with the current apparent field-of-view with an anti-movement rule. The method further includes determining that an alteration to the video stream generated by the camera device is required based on the comparison of the determined difference between the at least one attribute of the desired apparent field-of-view and the at least one attribute of the current apparent field-of-view with the anti-movement rule and, if so required, altering the video stream generated by the camera device to change from the current apparent field-of-view to the desired apparent field-of-view. Here, the generated crop-failure score is based on a location of one or more participants relative to the one or more boundaries of the current apparent field-of-view of a video stream generated by a camera device. Typically, the accumulated crop-failure score comprises a plurality of generated crop-failure scores.
Embodiments of the present disclosure may further provide a video streaming system which may be used to both monitor a physical environment to detect changes therein and provide a video stream of a portion of the physical environment to-be-transmitted to a remote viewer. Here, the video streaming system includes a software application which when executed by a processor prevents undesired and frequent adjustments to the boundaries of the video stream of the physical environment based on relatively minor detected changes therein. The video streaming system includes a camera device and a non-transitory computer readable medium that includes instructions which when executed by a processor are configured to perform a method. Generally, the camera device features an image sensor and a lens, where one or both of the lens and the image sensor are movable with respect to the other to change a focal length of the camera device and thus define an actual field-of-view of the physical environment. The method includes generating a crop-failure score for a current apparent field-of-view, comparing the crop-failure score or an accumulated crop-failure score to a crop-failure tolerance limit, and determining when to reframe the current apparent field-of-view. Determining when to reframe the current apparent field-of-view includes determining boundaries of a desired apparent field-of-view and altering the video stream when the crop-failure score or the accumulated crop-failure score is equal to or outside of the crop-failure tolerance limit. Here, the crop-failure score is based on a location of one or more participants in a portion of a physical environment relative to the boundaries of the current apparent field-of-view of a camera device. The accumulated crop-failure score generally comprises a plurality of generated crop-failure scores. The boundaries of a desired apparent field-of-view are based on the locations of the one or more participants in the portion of the physical environment. In some embodiments, the portion of the physical environment is defined by an actual field-of-view of the camera device. In some embodiments, the camera device comprises a processor for executing the method and the non-transitory computer readable medium having instructions for the method stored thereon. In other embodiments, the processor for executing the method and/or the non-transitory computer readable medium are comprised in a user device in communication with the camera device.
Embodiments of the present disclosure may further provide a video streaming system comprising a camera device, a non-transitory computer readable medium, and a processor. The camera device features an image sensor and a lens, wherein one or both of the lens and the image sensor are movable with respect to the other to change a focal length of the camera device and thus define an actual field-of-view of a physical environment. The non-transitory computer readable medium includes instructions which when executed by the processor are configured to perform a method. The method includes determining an actual composition of an apparent field-of-view of a to-be-transmitted video stream and comparing the actual composition of the apparent field-of-view to one or more composition rules. Typically, the actual composition comprises a location of one or more participants relative to one or more boundaries of the apparent field-of-view. The one or more composition rules may be based on general principles of photography. For example, the one or more composition rules may be based on one or a combination of negative space surrounding the one or more participants, negative space between the one or more participants and a boundary of the apparent field-of-view, asymmetry of the location of the one or more participants within the apparent field-of-view, a direction one or more participants are facing within the apparent field-of-view, a depth of field between at least two participants, a depth and/or width of the physical environment relative to the camera device, and a size, location, and/or type of one or more objects within the physical environment which are not participants. The method further includes determining a desired composition of the apparent field-of-view, determining that the desired composition is different from the actual composition, and based on a difference between the actual composition and the desired composition, or on an accumulation of differences between the desired composition and the actual composition, altering the to-be-transmitted video stream. Here, altering the to-be-transmitted video stream includes changing the boundaries of the apparent field-of-view to provide the desired composition. In some embodiments, the camera device comprises a processor for executing the method and the non-transitory computer readable medium having instructions for the method stored thereon. In other embodiments, the processor for executing the method and/or the non-transitory computer readable medium are comprised in a user device in communication with the camera device.
Embodiments of this disclosure may further provide a video conferencing method. The method includes generating a video stream of a current apparent field-of-view of a video conferencing environment, determining the boundaries of a desired apparent field-of-view of the video stream, determining a difference between at least one attribute of the desired apparent field-of-view with the current apparent field-of-view, and comparing the determined difference between the at least one attribute of the desired apparent field-of-view with the current apparent field-of-view with an anti-movement rule. The method further includes determining whether an alteration to the video stream generated by the camera device is required based on the comparison of the determined difference between the at least one attribute of the desired apparent field-of-view and the at least one attribute of the current apparent field-of-view with the anti-movement rule and, if so required, altering the video stream generated by the camera device to change from the current apparent field-of-view to the desired apparent field-of-view.
Embodiments of this disclosure may further provide a computer implemented method for monitoring the composition of a video stream of a physical environment. The method includes generating a video stream of an apparent field-of-view of a physical environment, generating one or more survey frames of at least a portion of an actual field-of-view of the physical environment least, and analyzing the one or more survey frames to generate survey data. Here, the video stream of the apparent field-of-view and the actual field-of-view are generated using the same camera device. Typically, generating the video stream of the apparent field-of-view includes analyzing the apparent field-of-view to determine an actual composition thereof, and comparing the actual composition to one or more composition rules to determine when to alter the video stream. Determining when to alter the video stream includes analyzing the survey data using the one or more composition rules, and altering the apparent field-of-view based on a difference between the actual composition and the desired composition. Generally, altering the apparent field-of-view includes one or a combination of panning, tilting, optically zooming, and digitally zooming the camera device.
Embodiments of this disclosure may further provide a video conferencing method for monitoring the composition of a video stream of a video conferencing environment, according to one embodiment. The method includes defining an actual field-of-view of the video conferencing environment, generating one or more survey frames that comprise at least a portion of the actual field-of-view, analyzing the one or more survey frames to generate survey data, generating a crop-failure score using the survey data, and altering the apparent field-of-view when the crop-failure score or an accumulated crop-failure score is equal to or outside of a crop-failure tolerance limit. Here, the crop-failure score is based on a location of one or more participants relative to one or more boundaries of an apparent field-of-view contained within a first video stream generated by the camera device, and wherein the apparent field-of-view is different from the actual field-of-view.
Embodiments of this disclosure may further provide a computer implemented method for monitoring the composition of a video stream of a physical environment and making adjustment thereto. The method desirably prevents frequent and undesirable adjustments to the video stream that might cause physical discomfort, such as motion related sickness, for the remote viewer. The method includes generating a video stream of an apparent field-of-view of a physical environment, generating one or more survey frames of at least a portion of an actual field-of-view of the physical environment least, and analyzing the one or more survey frames to generate survey data. Here, the video stream of the apparent field-of-view and the actual field-of-view are generated using the same camera device. Typically, generating the video stream includes analyzing the apparent field-of-view to determine an actual composition thereof, and analyzing the one or more survey frames or survey data to determine a desired composition of the apparent field-of-view. The method further includes determining a difference between one or more attributes of the actual composition and one or more attributed of the desired composition and comparing the difference to an anti-movement rule. The method further includes determining whether an adjustment to the apparent field-of-view should be made based on the comparison and altering the apparent field-of-view based on the determination. Altering the apparent field-of-view generally includes changing the boundaries of the apparent field-of-view to provide the desired composition.
So that the manner in which the above-recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective embodiments.
To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation.
Organizations of every kind have embraced video conferencing technology to fulfill at least a portion of their communication needs. As the proliferation of video conferencing systems increases so do user expectations related to the essential video and/or audio content exchanged between video conferencing sites, equipment cost, and usability. Thus, embodiments herein provide systems and software implemented methods that are configured to provide desired video conference content that is delivered at a lower cost compared to existing systems, and configured for seamless integration with readily available video conferencing software applications, such as Microsoft® Skype®, Apple® FaceTime® and applications available from Zoom® Video Communications, Inc. Some embodiments provided herein include systems and software that enable advanced auto-framing methods to be used during a video conference.
In one example, embodiments herein provide for digitally framing a portion of a video conferencing environment, such as about one or more initial or early arriving conference participants, to provide an apparent field-of-view used for a video data stream to-be-transmitted to a remote video conferencing endpoint. Concurrent therewith, embodiments herein provide for monitoring other areas of the video conferencing environment for changes that occur outside of the apparent field-of-view, such as late arriving conferencing participants. In some embodiments, the camera device used for automatically framing, generating, and transmitting the first apparent of view is the same camera device used to concurrently monitor the areas outside of the apparent field-of-view, thus providing an advanced single camera video conferencing system. As used herein, “advanced single camera video conferencing system” refers to a video conferencing system featuring a single camera device used for both framing the first apparent of view and monitoring areas outside of the apparent field-of-view. Use of the term “advanced single camera video conferencing system” is not meant to limit the use of other cameras with the video conferencing system for other purposes. Nor does the term limit the use of multiple advanced single camera video conferencing systems within the same designated physical space or to be used at the same time.
In some embodiments, the single camera device or system used with the methods set forth herein is a hybrid optical/digital camera capable of both optically zooming and digitally zooming. Optical zooming typically includes adjusting the focal length of a camera lens (“optical zoom lens”) where the focal length is the distance between the center of the optical zoom lens and an image sensor of the camera device. When the focal length is increased, i.e., the optical zoom lens is moved farther from the image sensor (or vice versa), a smaller portion of the desired scene strikes the image sensor resulting in magnification of the scene. The focal length also determines an actual field-of-view (also known as angle of view) that may be captured by the image sensor within the camera device. As the focal length is increased to optically zoom-in on a desired scene, the “actual field-of-view” of the camera narrows, thus optically cropping the periphery of the desired scene, and making objects therein seem closer. Thus, after performing an optical zoom, which may include either zooming-in or zooming-out, will create a new field-of-view (e.g., the smaller portion of the desired scene in a zooming-in case), which is then transmitted in a signal generated by the image sensor to a controller for processing and eventual delivery to one or more electronic components or devices. Optical zoom is generally considered lossless from a resolution point-of-view, meaning that the quality of the resulting optically zoomed image is the same as if the camera device had been moved closer to the desired scene.
A digitally zoomed image is typically made up of limited amount of data captured using a portion of the image sensor of the camera device. The limited amount of data is used to provide an image of an “apparent field-of-view.” The limited amount of data does not include data that might be otherwise acquired from the portions of the image sensor that are outside of, and thus excluded by, the apparent field-of-view. The amount of data may be limited simultaneously with the acquisition thereof, e.g., by only acquiring video data from the portion of the image sensor corresponding to the desired apparent field-of-view, or the data may be reduced using a post-processing digital zoom software application. Post-processing digital zoom typically uses a software algorithm to decrease a larger amount of data acquired from the “actual field-of-view” of the image sensor, such as by cropping an image captured by the image sensor from the actual field-of-view down to an area with the same aspect ratio as the original. Thus, digital zoom may be done simultaneously with the acquisition of the video data, by not including in the acquisition the undesired data from excluded regions of the image sensor, or may be done post processing after an original image, such as the actual field-of-view has been acquired by the image sensor. With either method, digital zoom effectively enlarges a scene to make it seem like the viewer is closer to the subject.
To compensate for the loss of resolution caused by enlarging an image, digital zoom of either method may or may not use image interpolation to appear to increase the number of pixels used to form the digitally zoomed image to those of the original image. Unfortunately, whether or not image interpolation is used, digital zoom is not lossless from a resolution point-of-view, meaning that the image quality deteriorates as the image is enlarged, and an unmodified digitally zoomed image may appear undesirably pixelated (e.g., without image interpolation) or undesirably blurry or smudgy (e.g., with image interpolation) when compared to the non-zoomed image or an image that has been captured using optical zoom. Thus, optical zoom is generally considered to provide superior image quality over an image created via a digital zoom.
Typically, those skilled in the art will prioritize optical zoom over digital zoom due to the relative image quality losses associated with the use of a digital zoom. Often this means that optical zoom will be used to the maximum extent possible towards achieving a desired image magnification before digital zoom is used to achieve additional image magnification if additional magnification is so desired. Nonetheless, in at least some of the video conferencing systems and methods described herein, digital zoom is counterintuitively prioritized over optical zoom at least to some extent. This means that digital zoom is used to crop and thus enlarge the desired scene, for example, by framing an apparent field-of-view about a number of initial participants, before switching to an optical zoom to maintain a desired minimum video image quality. Prioritizing digital zoom over optical zoom enables at least some of the methods set forth herein, such as by facilitating monitoring of a designated physical conference space outside of a framed video stream provided to a remote viewer using the same camera device as is used to capture the framed video stream. Such advanced single camera video conferencing systems, and methods related thereto, are further provided in the descriptions of the figures below.
The first video conferencing endpoint 102 features a designated physical space, such as the video conferencing environment 108, shown here as a conference room, having designated video conferencing equipment disposed therein. Here, the designated video conferencing equipment includes a user device 110, a microphone 112, a display 114, and a camera device 116.
Generally, the user device 110 represents any computing device capable of transmitting a video stream to a remote video conferencing device (e.g., the second video conferencing endpoint 104) over via the communication link 121 to the network 106. Examples of computing devices suitable for use as the user device 110 include, without limitation, laptops, personal computers, tablets, and smartphones. A processor 119 (e.g., CPU) within the user device 110 is used to execute one or more video conferencing software applications 118 capable of transmitting data (e.g., video data) received from the camera device 116 to the second video conferencing endpoint 104 via the communication link 121, and receiving video data from the second video conferencing endpoint 104, via the communication link 121. Examples of such video conferencing software applications, which may be used as the software application 118, include, without limitation, Microsoft® Skype®, Apple® FaceTime®, and applications available from Zoom® Video Communications, Inc. More generally, however, any video conferencing software application capable of receiving video data and transmitting the video data to a remote site can be used, consistent with the functionality described herein. In some embodiments, the user device 110 may be used to display video data captured at and received from the second video conferencing endpoint 104, on the display 114 (e.g., LED television, OLED television, plasma display). In some embodiments, the user device 110 (e.g., a laptop or tablet) be used to display video data captured at and received from the second video conferencing endpoint 104 on a native display of the user device 110. In some embodiments, the user device 110 is configured to execute on or more artificial intelligence (AI) applications 122, which may be used to perform aspects of the methods set forth herein.
In some embodiments, the video conferencing environment further includes a controller 124 disposed in wireless or wired communication with the camera device 116. The controller 124 may comprise any computing device capable of performing any one or combination of the same functions as are described below with respect to the controller 206 of the camera device 200. Examples of computing devices suitable for use as the controller 124 include, without limitation, laptops, personal computers, tablets, and smartphones. Here, the controller 124 includes a processor 126, a memory 128, and support devices 130. The processor 126 may be the same or substantially similar to the processor 212 as is described below for
As shown in
Herein, the memory 214, coupled to the processor 212, is non-transitory and represents any non-volatile type of memory of a size suitable for storing one or a combination of an operating system 222, one or more software applications 224, an environment structure 226, survey data 228, a crop-failure score generator 230, and an anti-movement rule 234. Examples of suitable memory that may be used as the memory 214 include readily available memory devices, such as random access memory (RAM), flash memory, a hard disk, or a combination of different hardware devices configured to store data. In some embodiments, the memory 214 includes memory devices external to the controller 206 and in communication therewith. In some embodiments, at least one of the one or more software applications 224 are executable by the processor 212 to perform one or more or combinations of the methods set forth herein, which advantageously simplifies integration with generally available video conferencing software applications 118. In other embodiments, one or more or combinations of the methods may be executed by one of a general-purpose computer or a custom electronic device external to the camera device 200 and in wired or wireless communication therewith. In some embodiments, the one or more software applications 224 stored in memory 214 include instructions which when executed by the processor 212 are configured to perform one or more of the methods described herein, such as at least a portion of the methods illustrated in and discussed in relation to
The video streaming device 218 is coupled to the processor 212 and is generally used to encode video data acquired from the image sensor 204 in a desired encoding format and at a desired bitrate. Generally, bitrate describes how much video data a video stream contains where higher resolution, higher frame rates, and lower compression requires an increased bitrate. In some embodiments, the video streaming device 218 and one or more of the software applications 224 executed by the processor 212 collectively function as a digital zoom feature of the camera device 200. In those embodiments, the video streaming device 218 is used to increase or decrease the apparent field-of-view of a video data stream acquired by the image sensor, i.e., digitally zoom-in to capture video data of only a portion of the actual field-of-view acquired by the image sensor. The digitally zoomed video data is typically transmitted to a user device, such as the user device 110 of
The communications device 220, communicatively coupled to the video streaming device 218, delivers the encoded video data to a user device, such as the user device 110 of
In other embodiments, which may be combined in any one of, combinations of, or combinations of portions of the methods set forth herein, video data is transmitted to a remote video conferencing endpoint(s) using conventional communication devices and protocols. For example, the video data may be transmitted to a remote video conferencing endpoint using a network interface card, Ethernet card, modem, wireless network hardware and/or other conventional computing device communication hardware.
At activity 310 the method 300 includes using a camera device, such as the camera device 116 or 200 of
Examples of considerations in setting the frame boundaries to define the first portion 402 may include, without limitation, room dimensions, seating area locations, and/or the location of presentation aids, such as a whiteboard. In some embodiments, the customized view (i.e., conference room framed view) may account for designated points of interest, such as the door 404, where it may be desirable to monitor the scene for activity, such as monitoring the door 404 for the entrance of new participants. In some embodiments, the customized view may account for designated points of distraction, such as the window 406 (
In some embodiments, the considerations described above, such as room dimensions, seating locations, locations of presentation aids, points of interest, such as the door 404, and points of distraction, such as the window 406, are stored in the memory 214 of the camera device 200 as the environment structure 226. In some embodiments, the initial frame boundaries that define the first portion 402 are manually set by a user at the beginning of a video conferencing session and thus override the initialization settings stored in the memory 214 of the camera device 200.
In some embodiments, activity 310 of the method 300 further includes capturing initial survey frames, e.g., one or more images, or a portion of a video data stream, having the framed boundaries that define the first portion 402, and performing an analysis on the initial survey frames to generate initial survey data. In some embodiments, performing an analysis on the initial survey frames to generate initial survey data includes performing a participant detection and/or recognition analysis to ascertain the number, and relative locations, of the conference participants within the first portion 402 of the video conferencing environment 400. In some embodiments, the initial survey data is stored in the memory of the camera device 200 for later retrieval and analysis.
In some embodiments, optically framing the first portion 402 of the video conferencing environment comprises using any one or combination of panning, tilting, and optically zooming the camera device 200 to establish a desired actual field-of-view, such as the first actual field-of-view 408 (
At activity 320, the method 300 includes digitally framing a second portion 410 (
In some embodiments, the controller 206 may use a participant detection and/or recognition software, such as an upper body detection software and/or a facial recognition software, to identify the location of conference participants and adjust the boundaries defining the second portion 410 accordingly. Typically, once the locations of conference participants are identified, the controller 206 will use settings stored in the memory thereof and/or encoded in the one or more software applications 224 to define the boundaries of the second portion 410 about the identified locations of conference participants. For example, the controller 206 may be used to apply general principles of photography, such as the rule of thirds, to define the white space about a conference participant or about a group of conference participants, to balance the composition of, and thus optimize, an apparent field-of-view of a video stream to-be-transmitted to remote conference participants.
In some embodiments, the boundaries defining the second portion 410 are determined using stored user preferences, and/or manually input by a user, such as by a conference participant using the user interface 208 set forth above in the description of
Digitally framing the second portion 410 of the video conferencing environment 400 provides an apparent field-of-view, such as the first apparent field-of-view 412 shown in
In some embodiments, the boundaries of the first apparent field-of-view 412, and thus the amount of the video conference room contained within the first apparent field-of-view 412, are limited by a desired threshold image resolution. The limit of the desired threshold image resolution ensures a desired minimum image quality of the transmitted field-of-view that is to-be-transmitted in the video stream delivered to the remote video conferencing endpoint. In some embodiments, the amount of available digital zoom is limited so as to prevent the image resolution from falling below a desired level, such as, for example, below a 720p or below a 1080p resolution. Thus, in some embodiments, the minimum resolution created after achieving a desired digital zoom level is, for example, 720p or 1080p. In some embodiments, the desired threshold image resolution is communicated to the processor 212 and/or the video streaming device 218 by the video conferencing software application 118 executing on the user device 110 so that a desired image resolution can be reached during activity 320. In general, once a desired digital framing configuration (e.g., digital zoom level) is selected by the controller 206, the data streamed from the camera device 200 will be maintained at that digital zoom level until there is need to make an adjustment in the field-of-view, as is discussed below. In some embodiments, the boundaries of the first apparent field-of-view 412, i.e., the amount of available digital zoom, are limited by the software application 224 that is being executed on the processor 212 based on the known resolution capabilities of the camera device 200. For example, depending on the resolution capability of the camera device, the digital zoom may be limited to about 4× or less, about 3.75× or less, about 3.5× or less, about 3.25× or less, about 3× or less, such as about 2.75× or less, about 2.5× or less, about 2.25× or less, or about 2× or less.
In some embodiments, the desired threshold image resolution is dynamically adjusted, by use of the processor 212 and one or more software algorithms stored in memory, based on changes in available bandwidth for transmitting the video stream to the user device 110 (
At activity 325, the method 300 optionally includes optically framing (or optically reframing) the actual field-of-view of the camera device 200 about a third portion 413 (
In some embodiments, such as shown in
Advantageously, activity 325 may be used after activity 320, or concurrently therewith, to balance the competing objectives of maintaining a relatively wide actual field-of-view to be used in activities 340 and 350 (set forth below) with ensuring the image resolution of the apparent field-of-view does not fall below user expectations for the video quality received at the remote video conferencing endpoint 104. A relatively wide actual field-of-view is facilitated by prioritizing digital zoom over optical zoom up to the point where the image quality (e.g., pixilation) of the apparent field-of-view falls below the desired threshold image resolution set forth above. Once it is determined that the image resolution of the apparent field-of-view is at risk of falling below the threshold image quality, then optical zoom may be used to achieve the desired boundaries of the apparent field-of-view. Thus, one will note that with activity 325, the actual field-of-view (i.e., larger or smaller FOV) of the camera device 200 will change from the original field-of-view, e.g., the first field-of-view 408 created during activity 310 due to the change in the optical zoom position within the camera device 200 between these activities. A method that may be used in combination with, or in place of, activity 325 is set forth in
At activity 330 the method 300 includes generating or capturing a video stream of the first apparent field-of-view 412. In some embodiments, generating the video stream of the first apparent field-of-view 412 includes using the video streaming device 218 of the camera device 200 to encode video data acquired from the image sensor 204.
In some embodiments, the video data acquired in activity 330 does not include data from the regions outside of the first apparent field-of-view, e.g., the excluded region 411 (
In other embodiments, the video stream of the first apparent field-of-view 412 may be acquired post-processing, e.g., by cropping the first apparent field-of-view 412 from video data acquired from an actual field-of-view. Post-processing digital zoom may be done using a software application 224, which is being executed on the processor 212 of the camera device 200 or a different software application executing on the user device 110 after a video stream of the first actual field-of-view 408 has been transmitted thereto.
Typically, the acquired video data is encoded into a desired encoding format, at a desired resolution, e.g., 720p, 1080p, 1440p, 4K (2160p), or 8K (4320p), and at a desired frame rate, e.g., in a range from about 30 frames per second (fps) to about 60 fps or more. In some embodiments, the desired encoding format, resolution, and frame rate are communicated to the controller 206 of the camera device 200 from a video streaming software application executing on a user device 110, for example, at the beginning of a video conferencing session using a handshake protocol. In some embodiments, the handshake protocol is between a video streaming device 218 of the controller 206 and the video conferencing software application 118. In some embodiments, one or both of the desired resolution and frame rate, and thus the bitrate of the video stream, are adjusted by the controller 206 based on changes in the bandwidth available for transmitting the video stream to the video conferencing software application 118 that is being executed on the user device 110, changed in the bandwidth available for transmitting the video stream to a remote video conferencing device for display, or both. A method that may be used to adjust the bitrate of the video stream based on available bandwidths for transmitting the video stream is set forth below in the description of
In some embodiments, activity 330 of the method 300 further includes transmitting the video stream to a video conferencing software application 118 that is being on the user device 110. In some embodiments, the video conferencing software application may perform further data processing methods on the video stream, e.g., by transcoding the received video data from a first format to a second format used by the video conferencing application, before transmitting the video stream to a remote video conferencing device for display.
At activity 340 the method 300 includes periodically generating survey frames of the actual field-of-view of the camera device 200. Here, the process of generating the video stream to-be-transmitted to a video conferencing software application that is being executed on a user device (i.e., activity 330) and generating survey frames of the current actual field-of-view (e.g., the first and/or second actual field-of-view 408, 415) to generate survey data are performed within the camera device 200. Typically, generating the to-be-transmitted video stream of the first apparent field-of-view 412 and generating the survey frames of the current actual field-of-view 408 or 415 are performed using the same image sensor 204 of the camera device 200.
Typically, each of the plurality of survey frames comprises a video frame of the actual field-of-view, or even a portion of the actual field-of-view, taken at different instants in time. In some embodiments, the survey frames may each include at least a portion of the first actual field-of-view 408 that is different from the first apparent field-of-view 412. For example, the survey frames may each contain at least a portion of the excluded region 411. In one example, the survey frames may each contain the first apparent field-of-view 412 and at least a portion of the excluded region 411. Herein, the plurality of survey frames are made up of a series of one or more survey frames periodically captured and simultaneously extracted from the video stream generated in activity 330 before the video stream is transmitted to the video conferencing software application executing on the user device. Thus, the to-be-transmitted video stream will generally not include the survey frames therein. In some embodiments, the survey data is stored in the memory of the camera device. A method 500 that may be used to periodically capture and simultaneously extract the survey frames from the video stream is set forth below
At activity 345, the method 300 includes analyzing each of the plurality of survey frames to generate survey data 228. Analyzing the survey frames may include using a software application 224 executing on the processor 212 to detect and/or recognize conference participants located within the actual field-of-view, e.g., by use of convolutional neural network. Typically, information comprising the number and locations of the conference participants within the boundaries of the actual field-of-view and/or with reference to the video conferencing environment structure 226 is stored in the memory 214 of the controller 206 as the survey data 228.
At activity 350, the method 300 includes detecting changes in the information found in the survey data 228 over time, such as the changes in the video conferencing environment 400 over time collectively illustrated in
At activity 360 the method 300 includes making an adjustment to the apparent field-of-view or actual field-field-of-view based on the analysis of the survey data performed in activity 350. Typically, as the number and/or locations of conference participants' changes during a video conferencing session, it will be desirable to again adjust the frame boundaries of the captured video stream. For example, based on the detected changes in the survey data, a fourth portion 414 (
In some embodiments, the activity 350 of the method 300 further includes determining which one of a plurality of conference participants is speaking and then in activity 360 adjusting the boundaries of the digitally framed field-of-view accordingly. For example, in some embodiments, facial detection and/or recognition analysis on the survey frames and/or the to-be-transmitted video stream and/or audio location data determined from the microphones 210 on the camera device 200 may be used to determine a specific conference participant is speaking, such as the participant speaker 418 shown in
In some embodiments, the activity 350 of the method 300 further includes monitoring a point of interest, such as the door 404, for a change in activity and adjusting the actual based thereon. For example, if a late arriving conference participant were to enter the video conferencing environment 400 through the door 404 and then disappear from the first actual field-of-view 408, the controller 206 of the camera device 200 might recognize that the late arrival is in the video conferencing environment 400 but outside of the actual (first) field-of-view 408 of the image sensor 204. Thus, during activity 360 the actual field-of-view may then be adjusted by optically reframing the video conferencing environment 108, e.g., optically zooming out, until the new actual field-of-view includes the new arrival's location. Once the new arrival is “found,” one or more activates of the method 300 can be repeated until the new conference participant is desirably included within an apparent field-of-view of the camera device 200.
In other embodiments, the method 300 further includes excluding points of distraction from the analysis of the survey data. For example, here it may be desirable to ignore changes in activity that are observable through the window 406 (
In embodiments herein, various activities of the method 300 may be repeated throughout a video conferencing session to continuously update a digitally framed field-of-view as new conference participants join (
Beneficially, aspects of the method 300, may be performed without the need to copy and store the to-be-transmitted video stream or perform other post processing operations thereon. Performing one or more aspects of the method 300 without the need for post processing allows for seamless integration with readily available video conferencing software applications 118 as the method 300 may be performed independent of the application and is thus invisible thereto. Typically, the video conferencing software application 118 can override the method 300, to control the optical and digital zoom features of the camera device 200 if such control is desired. In some embodiments, the method 300 is entirely performed using the controller 206 of the camera device 200, thus minimizing the bandwidth of the to-be-transmitted video stream for the benefit of wireless communication methods or any other communication method of limited bandwidth.
At activity 510, the method 500 includes framing a first portion of a video conferencing environment to provide a first actual field-of-view 408. Here, the first actual field-of-view 408 of
At activity 520 the method 500 includes generating a to-be-transmitted video stream 600 (
Herein, the to-be-transmitted video stream 600 does not include survey frames of the actual field-of-view 408, which are extracted (pulled) from the video stream 600 as described below.
At activity 530, the method 500 includes transmitting the video stream 600 to a video conferencing software application 118 that is being executed on a user device 110, such as described in the method 300.
At activity 540, the method 500 includes periodically generating, one or more survey frames of the actual field-of-view. In one example, the frames of the first actual field-of-view 408 are shown as survey frames 608a, 608b in
Generally, the frame rate capability of the camera device 200 will be equal to or greater than the frame rate requested by the video conferencing software application 118, e.g., a frame rate requested during a handshake protocol with the video streaming device 218 at the beginning of a video conferencing session. Thus, in some embodiments, generating the one or more survey frames of the first actual field-of-view 408 includes using the digital zoom feature of the camera device 200 to digitally zoom out, e.g., cease digitally cropping the actual field-of-view, between sequentially acquired frames of the to-be-transmitted video stream 600. For example, in some embodiments, the camera device 200 may be capable of providing a frame rate of about 40 frames per second (fps) or more, such as about 50 fps or more, or about 60 fps. In this example, the to-be-transmitted video stream 600 requested by the video conferencing software application 118 is less than the frame rate capabilities of the camera device 200 such as less than 60 fps, less than 50 fps, or less than 40 fps, or about 30 fps or less. In those embodiments, the camera device 200 may be used to digitally zoom out and acquire a frame of the first actual field-of-view 408 without interruption to the to-be-transmitted video stream 600, e.g., without changes to the time intervals between sequentially acquired frames of the apparent field-of-view(s) 412, 416.
In other embodiments, generating the one or more survey frames of the first actual field-of-view 408 includes using the digital zoom feature of the camera device 200 to digitally zoom out, e.g., cease digitally cropping the actual field-of-view for a limited number of sequentially acquired frames of the video stream 600. In those embodiments, generating the one or more survey frames of the actual field-of-view includes periodically interrupting the video stream of the apparent fields of view 412, 416.
One will note that using the digital zoom feature of the camera device 200 to generate even one survey frame may cause a disruption between generated frames of video data within a video stream if the survey frame is not removed therefrom. In other words, the field-of-view transmitted from the video streaming device 218 to the video conferencing software application 118 would briefly flicker from the desired apparent field-of-view to the first actual field-of-view 408 during the time the survey frame is being generated. Thus, to avoid any perceived discontinuity in the video stream 600, by a remote conference participant, the one or more captured frames of the first actual field-of-view 408, here the survey frames 608a, 608b are extracted (“pulled”) from the to-be-transmitted video stream 600 before the video stream 600 is transmitted to the video conferencing software application 118. In embodiments where generating the one or more survey frames includes periodically interrupting the view stream of the apparent fields of view 412, 416, the number of actual field-of-view frames that are generated are limited to a number that will not be perceived by a viewer.
At activity 550, the method 500 includes extracting the one or more generated frames of the first actual field-of-view 408, e.g., the survey frames 608a, 608b, from the video stream 600 before the video stream 600 is transmitted to the video conferencing application. Extracting the survey frames typically includes using a software application 224 that is being executed on the controller 206 of the camera device 200. For example, in some embodiments, the software application 224 executing on the controller 206 of the camera device 200 may instruct the video streaming device 218 to pull the generated survey frames 608a, 608b from the video stream to-be-transmitted to the video conferencing software application 118. The survey frames 608a, 608b may be stored in the memory 214 as survey data 228 for later use by a method set forth herein.
When used, the frequencies and duration of the periodic interruptions to generate the above-described survey frames 608a, 608b are selected to ensure that a remote conference participant will not perceive the loss of the frames (a gap) from the video stream. Typically, the loss of a single frame per second from a video stream of 30 frames per second would likely be imperceptible to a user and more frames could be extracted without detection by a user as the rate of frames per second is increased. In some embodiments, the rate of interruption and extraction of survey frames from the video stream may be, for example, more than 1 every 10 frames, no more than 1 every 20 frames, or no more than 1 every 30 frames. In some embodiments, a lost frame (e.g., extracted frames) can be compensated for by duplicating an immediately preceding frame of the video stream, which is then added into the video stream 600 in the lost frame's position.
In some embodiments, the generation and extraction of the survey frames may occur at regularly timed intervals, such as the time interval “t” of
At activity 560, the method 500 includes analyzing the one or more survey frames to generate survey data, such as described in activity 345 of the method 300. Typically, survey data is stored in the memory 214 of the camera device 200 where it may be later retrieved for further analysis, such as described in activity 350 of the method 300.
At activity 710, the method 700 includes optically framing, using a camera device 200, a first portion of a video conferencing environment to provide a first actual field-of-view. Optically framing the first portion of the video conferencing environment may include any one or combination of panning, tilting, and optically zooming the camera device. The level of the optical zoom created during activity 710 may be preset or selected based on an analysis of the conference room environment by the components within a controller 206 of the camera device 200.
At activity 720, the method 700 includes determining a desired threshold image resolution of a to-be-transmitted video stream of a desired apparent field-of-view. In some embodiments, the desired threshold image resolution is communicated to a controller of the camera device 200 during a handshake protocol between a video conferencing software application 118 and a video streaming device 218 of the controller 206. In some embodiments, the desired threshold image resolution is encoded in a software application 224 of the camera device 200. In some embodiments, the desired threshold image resolution is input and stored in the memory 214 of the camera device 200 as a user preference. In some embodiments, the desired threshold image resolution is set by a display resolution of a remote system (e.g., remote video conferencing system) that is configured to receive the to-be-transmitted video stream at a remote location. The display resolution information used to set the threshold image resolution can be input and stored in the memory 214 of the camera device 200 as a user preference or be received by the camera device 200 in a signal generated by the remote system. In some embodiments, the desired threshold image resolution is a dynamic value that changes based on available bandwidth for transmitting the video stream to the video conferencing software application 118, which is being executed on the user device 110 and/or the available bandwidth between different video conferencing endpoints. The desired apparent field-of-view is typically determined using the controller 206 of the camera device 200 to analyze one or more survey frames acquired from an actual field-of-view, e.g., to detect activity and/or the location of conference participants therein, such as described above in the method 300 of
At activity 730, the method 700 includes digitally zooming, using a digital zoom feature of the camera device 200, up to the desired threshold image resolution, to provide a digitally zoomed image.
At activity 740, the method 700 incudes comparing the digitally zoomed image to the desired apparent field-of-view to determine a difference therebetween. If the boundaries of the desired apparent field-of-view, as determined at activity 720, are the same as the boundaries of the digitally zoomed image the method 700 will end until a new or different apparent field-of-view is desired. In other words, if a change in activity or participants in the video conferencing environment is detected using the methods set forth above, the desired apparent field-of-view may change, and the method 700 will start again at activity 720. If the desired apparent field-of-view is larger than the digitally zoomed image, i.e., appears further away than the apparent field-of-view, then the method 700 will continue to activity 750.
Activity 750 of the method 700 includes optically zooming, using an optical zoom feature of the camera device, up to the desired apparent field-of-view, at which point the method 700 will end until a new or different apparent field-of-view or threshold image resolution is desired.
The method 700 beneficially balances the desire to maintain a relatively wide actual field-of-view, useful for monitoring the video conferencing environment, with ensuring the image resolution of the apparent field-of-view does not fall below user expectations for the video quality received at a remote conferencing endpoint. Maintaining the relatively wide actual field-of-view is facilitated by prioritizing digital zoom over optical zoom. To maintain image quality, digital zoom over optical zoom only up to the point where the image quality (e.g., pixilation) of the apparent field-of-view falls below the desired threshold image resolution set forth above. Once it is determined that the image resolution of the apparent field-of-view is at risk of falling below the threshold image quality, then optical zoom may be used to achieve the desired boundaries of the apparent field-of-view. Thus, one will note that the actual field-of-view (i.e., larger or smaller FOV) of the camera device 200 may change during the implementation of the method 700.
The methods set forth herein advantageously provide for continuous monitoring and optimization of an apparent field-of-view of a video stream to-be-transmitted to remote conference participants for the viewing benefit thereof. Nonetheless, it is recognized that frequent adjustments to the boundaries of the apparent field-of-view could be undesirably annoying and/or disorienting to a remote viewer. Thus, in some embodiments, adjustments to the field-of-view are controlled using the method 800 illustrated in
At activity 810 the method 800 includes identifying the locations of one or more conference participants within an actual field-of-view of a camera device. In this activity, a software algorithm running within the camera device 116, camera device 200, or controller 124, for example, analyzes data within a generated survey frame to detect conference participants within the actual field-of-view and to identify the locations thereof. Analyzing the survey frames may include using a software application 224 executing on the processor 212 to detect and/or recognize conference participants located within the actual field-of-view, e.g., by use of convolutional neural network.
At activity 820 the method 800 includes determining the locations of each of the one or more conference participants relative to the boundaries of an apparent field-of-view of the camera device 200 to generate one or more crop-failure scores. Here, each crop-failure score indicates the undesirability of a boundary of the current apparent field-of-view, i.e., the amount of crop or cropping, relative to the location of the corresponding conference participant. Generally, a relatively high crop-failure score indicates that the crop defining a boundary of the current apparent field-of-view relative to a conference participant is highly undesirable (e.g., a noticeable and/or significant portion of the participant is cropped) while a null crop-failure score indicates that the crop defining the boundary of the relative to the conference participant is ideal (e.g., a small portion or no portion of the participant is cropped and the location of the participant within the apparent field-of-view has a desirable symmetry). In one example, a relatively high crop-failure score may be assigned when an individual conference participant is located completely outside of a boundary of the current apparent field-of-view and a slightly lower score (but still relatively high) may be assigned if only a portion of the participant is located outside of the boundary. In another example, a relatively low crop-failure score may be assigned if a conference participant has shifted slightly in their chair so that the white space defined about the conference participant is no longer of an ideal composition, e.g., the conference participant is no longer centered within the current apparent field-of-view.
In some embodiments, the crop-failure scores are determined using a using a software application, such as the crop-failure score generator 230 executing on the processor 212, to assign a crop-failure score selected from a group of crop-failure scores which have been predetermined based on likely failure events. For example, crop-failure scores may be predetermined for failure events such as a participant located outside of the apparent field-of-view or a portion of a participant located outside of the apparent field-of-view. When the failure event includes a portion of a participant located outside of the apparent field-of-view, a corresponding crop-failure score may be determined by analyzing what portion of the participant's body falls outside of the apparent field-of-view. For example, a failure event where an upper boundary of an apparent field-of-view crops all or a portion of a participant's head may generate a higher crop-failure score than a crop-failure score which is generated from a failure event where a left or right boundary of the apparent field-of-view crops all or a portion of a participant's shoulder.
Typically, at least one crop-failure score is generated for each conference participant identified at activity 810. In some embodiments, crop-failure scores for an individual conference participant are generated for each of the boundaries defining the current apparent field-of-view. For example, for the generally rectangular apparent fields-of-view illustrated herein, up to four crop-failure scores may be generated for each individual conference participant each time a survey frame is analyzed at activity 810. Thus, if two conference participants are detected in the data from a survey frame at activity 810, up to eight crop-failure scores may be generated for that survey frame. If the number of conference participants remains the same, and a survey frame is generated every one second, then up to eighty crop-failure scores may be generated in a 10 second time period. Here, the individual crop-failure scores are stored in the memory 214 of the controller 206 as the crop-failure data 232, which is also referred to herein crop data or field-of-view data.
At activity 830 the method 800 includes comparing the crop-failure score or an accumulation of the crop-failure scores to a predetermined crop-failure tolerance limit, which is stored in the memory 214. The crop-failure tolerance limit may be a lower limit or an upper limit. In some embodiments, the crop-failure score or the accumulation of crop-failure scores may be compared to both lower and upper tolerance limits. Whether the crop-failure tolerance limit(s) is a lower limit, or an upper limit, or both may be based on the numerical values of crop-failure scores assigned to corresponding crop-failure events and/or the method used to determine the accumulated crop-failure scores. For example, if the crop-failure scores are positive integers and/or fractions, and the method used to determine an accumulated crop-failure score is a summation of the positive integers and/or fractions the crop-failure tolerance limit will likely be an upper limit. Thus, crop-failure scores and accumulated crop-failure scores that are greater than an upper crop-failure tolerance limit will be “outside of the upper limit.” Similarly, if the crop-failure scores are negative integers and/or fractions and the method of determining the accumulated crop-failure score is summation, then the crop-failure tolerance limit will likely be a lower limit. Thus, crop-failure scores and accumulated crop-failure scores that are less than a lower crop failure tolerance limit are “outside of the lower limit.” One should note that these examples are not intended to be limiting as it is recognized that any number or combination of schemes of assigning the crop-failure scores and/or mathematically manipulating a plurality or crop-failure scores to determine an accumulated crop-failure score may be used with the methods set forth herein. In some embodiments, the crop-failure tolerance limit(s) may change based on the zoom level, optical or digital, of the camera device 200 and/or based on the stability (shakiness) of the image provided in the video stream.
Here, one or more accumulated crop-failure scores are generated using crop-failure data 232, which comprises individual crop-failure scores and/or other information calculated therefrom. In some embodiments, the crop-failure data 232 comprises individual crop-failure scores acquired within a rolling time window where crop-failure scores that are older than the time period defined by the rolling time window will decay or drop from the crop-failure data 232 as new crop-failure scores are acquired. In some embodiments, an accumulated crop-failure score may comprise a sum of the individual crop-failure scores acquired within the rolling time window or rolling number of scores (e.g., sum of the last three generated scores). In some embodiments, individual accumulated crop-failure scores are separately generated for each of the boundaries defining a current apparent field-of-view. For example, for the rectangular apparent fields-of-view illustrated, each having four boundaries, a corresponding four accumulated crop-failure scores may be generated.
Generally, if at least one of the one or more crop-failure scores or accumulated crop-failure scores are equal to or are outside of the crop-failure tolerance limit, then the method 800 will continue to activity 840. If none of the one or more accumulated crop-failure scores are equal to or are outside of the crop-failure tolerance limit, the method 800 will return to activity 810 to identify the locations of one or more conference participants within the next generated survey frame of the actual field-of-view. Beneficially, the accumulated crop-failure scores generated in activity 830 prevent a software application executing on the controller 206 from making unnecessary and/or frequent requests to adjust the actual and/or apparent field-of-views. For example, a relatively high scoring failure event such as a newly detected individual located outside of the apparent field-of-view may be fleeting if the newly detected individual quickly enters and leaves the video conferencing environment. In this example, it would be undesirable to adjust the apparent field-of-view as the newly detected individual enters the conference room only to adjust it back a moment later when the newly detected individual leaves the conference room. Advantageous, the rolling accumulated crop-failure scores described above may never reach the threshold value as the number of crop-failure scores assigned to the newly detected individual would be limited by the number of survey frames acquired during the short time period the individual was in the video conference environment.
At activity 840, the method 800 includes comparing a difference between the current apparent field-of-view and a desired apparent field-of-view. The desired apparent field-of-view may be determined using the one or more software applications 224 executing on the controller 206 to define the boundaries of a desired apparent field-of-view based on the locations of the one or more conference participants identified at activity 810. In some embodiments, the difference between the current apparent field-of-view and the desired apparent field-of-view is analyzed using an anti-motion sickness software application, which may be stored in the memory 214 and executed by the processor 212. Typically, the anti-motion sickness software application is used to determine a difference between at least one attribute of the desired apparent field-of-view and at least one attribute of the current apparent field-of-view. The determined difference is then compared to the anti-movement rule 234.
At activity 850 the method 800 includes determining whether an alteration to the current apparent field-of-view should be allowed. The determination of whether to allow the alteration to the current apparent field-of-view is based on the comparison of the difference between the at least one attribute of the current apparent field-of-view and the at least one attribute of the desired apparent field-of-view to the anti-movement rule. For example, in some embodiments, a software application that is being executed by the processor 212 applies the anti-movement rule 234 to prevent adjustments that would be made to the apparent field-of-view and/or the actual field-of-view that might cause a viewer of the video stream to experience unwanted physical effects. In one example, the application of the anti-movement rule 234 will prevent changes to the video stream that would be perceived by a user as video flicker and/or a perceived swaying motion of portions of the field-of-view within the video stream, which would cause motion sickness.
Generally, in determining whether the camera device 200 is allowed to change the current apparent field-of-view, the anti-motion sickness software application will apply the anti-movement rule 234 to analyze the differences between various attributes of the current apparent field-of-view and the desired apparent field-of-view, such as the magnitude of the change in the field-of-view between the current apparent field-of-view and the desired apparent field-of-view (e.g., percentage change in viewing area, amount of pan, tilt or zoom adjustment, the symmetry of the one or more participants within the frame, etc.), the frequency of adjustments being made to the apparent field-of-view (e.g., related to amount of time since last change, number of changes within a period of time, etc.), and/or the effect the change will have on the composition of the current apparent field-of-view (e.g., change in number of conference participants, does the field-of-view include a person that is speaking, etc.), and the whether the composition of the current apparent field-of-view is generally satisfactory, even if it is not necessarily optimal. In some embodiments, the anti-motion sickness software, applying the anti-movement rule 234, will analyze and compare the difference between the at least one attribute of the current apparent field-of-view and the desired apparent field-of-view to determine if the camera device 200 is allowed to change the current apparent field-of-view. In some embodiments, the anti-movement rule 234 includes one or more logical expressions, defined by software instructions stored in memory, that allow the anti-motion sickness software to determine that a change should be made to the apparent field-of-view based on an analysis of an attribute of the current apparent field-of-view and the desired apparent field-of-view and the anti-movement rule. In some embodiments, the anti-movement rule simply includes one or more anti-motion threshold values that are used to determine if a change should be made to the apparent field-of-view by comparing an anti-motion threshold value with a determined difference between an attribute of the current apparent field-of-view and the desired apparent field-of-view.
In one example of the processes performed in activity 850, if the frequency in time of the requested changes is too high (i.e., frequency that a crop-failure score or the accumulated crop-failure score has reached or exceeded the threshold value is too high), the anti-movement rule 234 may be used to disallow or prevent the requested change from occurring. In another example, if the magnitude of the requested change is relatively low, the anti-movement rule 234 may be used to disallow or prevent the requested change from occurring. In yet another example, if the magnitude of the change requires only a small adjustment in the amount of zoom perceived by the viewer, such as about 50% or less, about 40% or less, about 30% or less, about 20% or less, or about 10% or less, and the composition of the current apparent field-of-view is generally satisfactory, for example, the upper bodies of the participants are within the boundaries of the current apparent field-of-view, the anti-movement rule 234 may be used to reject the request to change the current apparent field-of-view. In some embodiments, the magnitude of the requested change may be measured by the degree of pan and/or tilt required of the camera device 200 to provide the desired apparent field-of-view. For example, if the degree of pan and/or tilt requested is about 5 degrees or less, such as about 4 degrees or less, about 3 degrees or less, about 2 degrees or less, or about 1 degree or less, and the composition of the current apparent field-of-view of view is generally satisfactory, the anti-movement rule 234 may be used to reject the requested change to the apparent field-of-view.
If the anti-movement rule 234 allows the requested change to the current apparent field-of-view the method 800 will continue to activity 860. If the anti-movement rule 234 rejects the requested change the method 800 will return to activity 810 to identify the locations of one or more conference participants within the next generated survey frame of the actual field-of-view.
At activity 860, the anti-motion sickness software application, based on the requested desired apparent field-of-view of view determined in the prior activities, will then provide a control signal to the components within the camera device, which are used to adjust the apparent field-of-view, or actual field-of-view. The control signal can include parameters that will cause the camera device to alter its pan, tilt and/or zoom in a manner that will provide the desired apparent field-of-view of the conference room environment without causing a user that is viewing the changing video stream to experience unwanted physical effects due to the camera adjustment. At activity 860 the method 800 includes altering the apparent field-of-view to correspond to the boundaries defined by the desired apparent field-of-view, thus providing a new current apparent field-of-view. Once the apparent field-of-view has been altered, the accumulated crop-failure scores may be reset to a null value, and the method 800 will return to activity 810.
In some embodiments, the activities 810-860 of method 800 may be serially repeated multiple times. In some embodiments, one or more of the activities 810-860 of method 800 may be left out during the execution of the method 800. In one example, activities 810, 840, or 850, or 840 and 850 may be omitted from at least one pass of multiple repeated passes through the activities performed in method 800.
In some embodiments, one or both of the crop-failure score generator 230 and the anti-movement rule 234 are generated using a machine learning AI algorithm (hereafter AI). Generally, the AI is trained using viewer input collected before, during, and/or after actual or simulated video conferencing sessions. Examples of information that may be collected from conference participants as viewer input include, without limitation, annoyance, fatigue, attentiveness, and disorientation, e.g., motion sickness. Viewer input may be collected before, during, and after the video conferencing session and may include both real-time reactions and accumulated responses. For example, viewer input may include brief annoyances at changes to the apparent field-of-view that the viewer perceives as unnecessary, such as due to participants briefly engaging in a side conversation and/or increasing frustration (accumulated annoyance) at a frequency of changes to the apparent field-of-view over the course of a video conferencing session.
In some embodiments, the AI may be trained using biological sensor information in addition to, or in place of, the viewer input. Biological sensor information typically includes any information which may be collected using a device configured to measure a physiological response (or lack thereof) to changes in the viewer experience over the course of a video conferencing session. Examples of sensors that may be used to train the machine learning AI include, without limitation, electroencephalogram EEG, electrocardiogram ECG, picopicogram PPG, galvanic skin response GSR, and/or wearable sensors used to detect a motion sickness response, such as nausea.
In some embodiments, the machine learning AI correlates the viewer experience information, e.g., the viewer input and/or biological response, to changes in the apparent field-of-view resulting from panning, tilting, and/or zooming (optically and/or digitally) the camera device 200 using the methods set forth herein. The correlation is used to generate or update the crop-failure score generator 230 and/or the anti-movement rule 234, which may be stored in the memory 214 of the camera device 200. The crop-failure score generator 230 may be used to determine the crop-failure scores applied for specific crop failure events during the method 800. The anti-movement rule 234 may be used to determine a viewer's likely response to a change in the apparent field-of-view based on a detected change in the video conferencing environment, such as a response to the perceived movement of the camera device from panning, tilting, and/or zooming the camera device 200. Based on a viewer's likely response as determined by the application of the anti-movement rule 234, the controller 206 may adjust the frequency, speed, and/or perceived movement of the camera device 200 when responding to detected changes within a video conferencing environment.
Beneficially, the method 800 may be used with any one or combination of the methods and devices described herein to improve the viewer experience by minimizing unnecessary, annoying, or frequent repositioning of the camera device as well as eliminating undesirable physical effects that may be associated therewith. In some embodiments, updated crop-failure score generator 230 and/or anti-movement rule 234 are periodically delivered to the camera device 200 via device-ware updates to beneficially provide users with a continuously improving video conferencing experience over the lifetime of a camera device.
At activity 910 the method 900 includes generating one or more survey frames of the physical environment, such as a video conferencing environment, where each of the one or more survey frames comprise at least a portion of an actual field-of-view of a camera device 200. In some embodiments, the one or more survey frames are generated between sequentially generated frames of an apparent field-of-view, which will make up the video stream to-be-transmitted to a user device 110. The processes performed in activity 910 may include one or more of the processes described above in relation to
In some embodiments, the method 900 further includes transmitting the video stream to the user device 110. Typically, the one or more survey frames are extracted from the video stream before the video stream is transmitted to the user device 110. In some embodiments, a video conferencing software application that is being executed on the user device 110 is configured to receive the video stream and transmit the video stream to a remote video conferencing device for display.
At activity 920, the method 900 includes analyzing the one or more survey frames to generate survey data 228, such as described above in activity 345 of the method 300.
At activity 930 the method 900 includes generating a video stream of an apparent field-of-view by use of the camera device 200. Typically, the apparent field-of-view is of a portion of the actual field-of-view and is achieved by using a digital zoom feature of the camera device, as discussed above. Here, generating the video stream of the apparent field-of-view includes analyzing the apparent field-of-view, as set forth in activities to determine an actual composition thereof and comparing the actual composition of the apparent field-of-view to one or more composition rules as respectively set forth in activities 940 and 950.
Here, at least some of the one or more composition rules may be based on general principles of photography, such as one or more of the negative space surrounding the one or more objects, the negative space between the one or more objects, and a boundary of the apparent field-of-view, the asymmetry of the location of the one or more objects within the apparent field-of-view, a direction one or more objects are facing within the apparent field-of-view, a depth of field between at least two objects, and a ratio of a depth to width of the physical environment relative to the camera device. When the objects are conference participants, the one or more composition rules may include any one of, or a combination of, negative space surrounding the one or more participants, negative space between the one or more participants and a boundary of the apparent field-of-view, asymmetry of the location of the one or more participants within the apparent field-of-view, a direction one or more participants are facing within the apparent field-of-view, a depth of field between at least two participants, and a ratio of a depth to width of the physical environment relative to the camera device.
At activity 960, the method 900 includes analyzing the survey data using the one or more composition rules to determine a desired composition of the apparent field-of-view and altering the apparent field-of-view based on a difference between the actual composition and the desired composition. In some embodiments, the difference between the actual composition and the desired composition is based on a change in a location of one or more objects within the physical environment between one or more generated survey frames. In embodiments where the physical environment is a video conferencing environment, the objects may include conference participants.
In some embodiments, method 900 further includes generating a crop-failure score based on the comparison of the actual composition of the apparent field-of-view to the one or more composition rules and comparing the crop-failure score or an accumulated crop-failure score to a crop-failure tolerance limit. Here, the accumulated crop-failure score comprises a plurality of generated crop-failure scores. Typically, in those embodiments, the apparent field-of-view is not altered until the crop-failure score, or the accumulated crop-failure score is equal to or outside of the crop-failure tolerance limit.
Generally, generating the crop-failure score includes comparing a location of one or more participants relative to one or more boundaries of the apparent field-of-view to a plurality of predetermined crop failure events, each having a corresponding predetermined crop-failure score. The one or more crop-failure scores may be generated for the respective locations of at least one of the one or more participants based on the comparison to the plurality of predetermined crop-failure events. In some embodiments, the accumulated crop-failure score comprises a plurality of generated crop-failure scores taken within a rolling time window, and individual crop-failure scores acquired outside of the rolling time window are not included in the determination of the accumulated crop-failure score.
At activity 970, the method 900 includes altering the apparent field-of-view based on a difference between the actual composition and the desired composition. In some embodiments, the apparent field-of-view is altered at activity 970 after comparing the difference between the actual composition and the desired composition with the anti-movement rule 234, and then a determination is made, based on the comparison, that the alteration to the apparent field-of-view should be made. Altering the apparent field-of-view may include any one or a combination of panning, tilting, optically zooming, and digitally zooming the camera device 200. In some embodiments, altering the apparent field-of-view also alters the actual field-of-view.
Beneficially, the method 900 may be used with any one or combination of the methods and devices described herein to improve a viewer's experience by adjusting the composition of a video stream based on changes therein. Advantageously, the method dampens the effect of automatic adjustments by minimizing unnecessary, annoying, or frequent repositioning of the camera device as well as eliminating undesirable physical effects that may be associated therewith.
While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
This application is a Continuation of co-pending U.S. patent application Ser. No. 17/136,832 filed on Dec. 29, 2020, which is a divisional of U.S. patent application Ser. No. 16/833,854, filed on Mar. 30, 2020, now patented as U.S. Pat. No. 10,972,655, patented on Apr. 6, 2021, both Applications are herein incorporated by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5434617 | Bianchi | Jul 1995 | A |
| 6069655 | Seeley et al. | May 2000 | A |
| 6240303 | Katzur | May 2001 | B1 |
| 6353461 | Shore | Mar 2002 | B1 |
| 6392694 | Bianchi | May 2002 | B1 |
| 6519648 | Eyal | Feb 2003 | B1 |
| 6611281 | Strubbe | Aug 2003 | B2 |
| 6731334 | Maeng et al. | May 2004 | B1 |
| 6803945 | Needham | Oct 2004 | B1 |
| 6829391 | Comaniciu et al. | Dec 2004 | B2 |
| 6995794 | Hsu et al. | Feb 2006 | B2 |
| 7133513 | Zhang | Nov 2006 | B1 |
| 7298733 | Sakai et al. | Nov 2007 | B2 |
| 7349008 | Rui et al. | Mar 2008 | B2 |
| 7433327 | Klarville et al. | Oct 2008 | B2 |
| 7746401 | Wu et al. | Jun 2010 | B2 |
| 7876923 | Finnegan et al. | Jan 2011 | B2 |
| 7940432 | Shih et al. | May 2011 | B2 |
| 8094193 | Peterson et al. | Jan 2012 | B2 |
| 8237764 | Chen et al. | Aug 2012 | B1 |
| 8244103 | Shore | Aug 2012 | B1 |
| 8284254 | Romanowich et al. | Oct 2012 | B2 |
| 8358328 | Friel et al. | Jan 2013 | B2 |
| 8368752 | Lin et al. | Feb 2013 | B2 |
| 8396007 | Gonia et al. | Mar 2013 | B2 |
| 8471889 | Lee et al. | Jun 2013 | B1 |
| 8520072 | Slavin | Aug 2013 | B1 |
| 8547414 | Sheeley | Oct 2013 | B2 |
| 8659638 | Chao et al. | Feb 2014 | B2 |
| 8744125 | Zhu | Jun 2014 | B2 |
| 8780168 | Corley et al. | Jul 2014 | B2 |
| 8842161 | Feng et al. | Sep 2014 | B2 |
| 8872882 | Shanmukhadas et al. | Oct 2014 | B2 |
| 8885057 | Mock | Nov 2014 | B2 |
| 8913103 | Sargin et al. | Dec 2014 | B1 |
| 9001183 | Mauchly | Apr 2015 | B2 |
| 9077906 | Tsai et al. | Jul 2015 | B1 |
| 9125138 | Abuan et al. | Sep 2015 | B2 |
| 9125146 | Edara et al. | Sep 2015 | B1 |
| 9197856 | Tangeland et al. | Nov 2015 | B1 |
| 9237307 | Vendrow | Jan 2016 | B1 |
| 9270941 | Lavelle | Feb 2016 | B1 |
| 9286693 | Yoo | Mar 2016 | B2 |
| 9307191 | Berrett et al. | Apr 2016 | B2 |
| 9313556 | Borel et al. | Apr 2016 | B1 |
| 9338395 | Wang et al. | May 2016 | B2 |
| 9547678 | Wnuk | Jan 2017 | B2 |
| 9549153 | Delorenzi et al. | Jan 2017 | B1 |
| 9763029 | Mirza et al. | Sep 2017 | B2 |
| 9785772 | Johansson et al. | Oct 2017 | B1 |
| 9798933 | Meisser et al. | Oct 2017 | B1 |
| 9801234 | Caine et al. | Oct 2017 | B2 |
| 9805567 | Borel et al. | Oct 2017 | B2 |
| 9942661 | Dusse et al. | Apr 2018 | B2 |
| 10115396 | Anderson et al. | Oct 2018 | B2 |
| 10268759 | Witt et al. | Apr 2019 | B1 |
| 10341792 | Zhang et al. | Jul 2019 | B1 |
| 10360457 | Meisser et al. | Jul 2019 | B2 |
| 10904446 | Ostap et al. | Jan 2021 | B1 |
| 10951858 | Ostap et al. | Mar 2021 | B1 |
| 10965908 | Ostap et al. | Mar 2021 | B1 |
| 10972655 | Ostap et al. | Apr 2021 | B1 |
| 20020106137 | Chen et al. | Aug 2002 | A1 |
| 20040003409 | Berstis | Jan 2004 | A1 |
| 20040027242 | Venetianer et al. | Feb 2004 | A1 |
| 20040080627 | Kroll | Apr 2004 | A1 |
| 20050014490 | Desai et al. | Jan 2005 | A1 |
| 20050160457 | Rui et al. | Jul 2005 | A1 |
| 20050168574 | Lipton et al. | Aug 2005 | A1 |
| 20050271251 | Russell | Dec 2005 | A1 |
| 20050289224 | Deslippe et al. | Dec 2005 | A1 |
| 20060009985 | Ko et al. | Jan 2006 | A1 |
| 20060069797 | Abdo et al. | Mar 2006 | A1 |
| 20060095339 | Hayashi et al. | May 2006 | A1 |
| 20070002141 | Lipton et al. | Jan 2007 | A1 |
| 20070024706 | Brannon et al. | Feb 2007 | A1 |
| 20070048712 | Plastina et al. | Mar 2007 | A1 |
| 20080018737 | Suzuki et al. | Jan 2008 | A1 |
| 20080021879 | Cheng | Jan 2008 | A1 |
| 20080089268 | Kinder et al. | Apr 2008 | A1 |
| 20080100704 | Venetianer et al. | May 2008 | A1 |
| 20080192666 | Koskan et al. | Aug 2008 | A1 |
| 20090015658 | Enstad et al. | Jan 2009 | A1 |
| 20090031381 | Cohen | Jan 2009 | A1 |
| 20090141939 | Chambers et al. | Jun 2009 | A1 |
| 20090174530 | Yen et al. | Jul 2009 | A1 |
| 20090219300 | Peleg et al. | Sep 2009 | A1 |
| 20090284579 | Knaz | Nov 2009 | A1 |
| 20090298420 | Haartsen et al. | Dec 2009 | A1 |
| 20100082585 | Barsook et al. | Apr 2010 | A1 |
| 20100092037 | Peleg et al. | Apr 2010 | A1 |
| 20100135643 | Fleming | Jun 2010 | A1 |
| 20100284389 | Ramsay et al. | Nov 2010 | A1 |
| 20100308765 | Moore et al. | Dec 2010 | A1 |
| 20100315497 | Jiang | Dec 2010 | A1 |
| 20110050843 | Cheng et al. | Mar 2011 | A1 |
| 20110055256 | Phillips et al. | Mar 2011 | A1 |
| 20110074570 | Feldstein | Mar 2011 | A1 |
| 20110099493 | Yu et al. | Apr 2011 | A1 |
| 20110116538 | Chuang et al. | May 2011 | A1 |
| 20110128350 | Oliver et al. | Jun 2011 | A1 |
| 20110129048 | Barbe et al. | Jun 2011 | A1 |
| 20110131498 | Chao et al. | Jun 2011 | A1 |
| 20110141314 | Liu et al. | Jun 2011 | A1 |
| 20110148759 | Hwang et al. | Jun 2011 | A1 |
| 20110148792 | Hwang et al. | Jun 2011 | A1 |
| 20110158441 | Batra | Jun 2011 | A1 |
| 20110161836 | Mu et al. | Jun 2011 | A1 |
| 20110197214 | Burton et al. | Aug 2011 | A1 |
| 20110223893 | Lau et al. | Sep 2011 | A1 |
| 20110285842 | Davenport et al. | Nov 2011 | A1 |
| 20120019611 | Wu et al. | Jan 2012 | A1 |
| 20120169883 | Chang et al. | Jul 2012 | A1 |
| 20120170802 | Millar | Jul 2012 | A1 |
| 20120223960 | Chiang et al. | Sep 2012 | A1 |
| 20120268626 | Lu et al. | Oct 2012 | A1 |
| 20120308077 | Tseng | Dec 2012 | A1 |
| 20130007499 | Moy | Jan 2013 | A1 |
| 20130024891 | Elend | Jan 2013 | A1 |
| 20130174223 | Dykeman et al. | Jul 2013 | A1 |
| 20130183958 | Wesby | Jul 2013 | A1 |
| 20130194378 | Brown | Aug 2013 | A1 |
| 20130329003 | Hsia et al. | Dec 2013 | A1 |
| 20130335508 | Mauchly | Dec 2013 | A1 |
| 20140012990 | Ko | Jan 2014 | A1 |
| 20140013228 | Hutten | Jan 2014 | A1 |
| 20140043485 | Bateman et al. | Feb 2014 | A1 |
| 20140043493 | Bateman et al. | Feb 2014 | A1 |
| 20140043495 | Bateman et al. | Feb 2014 | A1 |
| 20140049595 | Feng et al. | Feb 2014 | A1 |
| 20140092001 | Kikkawa | Apr 2014 | A1 |
| 20140092201 | Williams | Apr 2014 | A1 |
| 20140111600 | Schaefer et al. | Apr 2014 | A1 |
| 20140115115 | Kuang | Apr 2014 | A1 |
| 20140176708 | Ramakrishnan | Jun 2014 | A1 |
| 20140178043 | Kritt et al. | Jun 2014 | A1 |
| 20140186026 | Oshima et al. | Jul 2014 | A1 |
| 20140213227 | Rao | Jul 2014 | A1 |
| 20140274203 | Ganong, III et al. | Sep 2014 | A1 |
| 20140313282 | Ma et al. | Oct 2014 | A1 |
| 20140313346 | Huang et al. | Oct 2014 | A1 |
| 20140355907 | Pesavento et al. | Dec 2014 | A1 |
| 20150015735 | Rav-Acha et al. | Jan 2015 | A1 |
| 20150022636 | Savransky | Jan 2015 | A1 |
| 20150091737 | Richardson | Apr 2015 | A1 |
| 20150103179 | Galvin | Apr 2015 | A1 |
| 20150109399 | Kuscher et al. | Apr 2015 | A1 |
| 20150110259 | Kaye et al. | Apr 2015 | A1 |
| 20150156257 | Li et al. | Jun 2015 | A1 |
| 20150189402 | Outtagarts et al. | Jul 2015 | A1 |
| 20150208352 | Backholm et al. | Jul 2015 | A1 |
| 20150254435 | Fells | Sep 2015 | A1 |
| 20150256577 | Gutierrez Vilaro et al. | Sep 2015 | A1 |
| 20150289295 | Granbery | Oct 2015 | A1 |
| 20150296187 | Baldwin | Oct 2015 | A1 |
| 20150326638 | Yarygin | Nov 2015 | A1 |
| 20150350611 | Pearson et al. | Dec 2015 | A1 |
| 20150379021 | Kuper | Dec 2015 | A1 |
| 20160012609 | Laska | Jan 2016 | A1 |
| 20160027262 | Skotty | Jan 2016 | A1 |
| 20160055381 | Adsumilli | Feb 2016 | A1 |
| 20160057385 | Burenius | Feb 2016 | A1 |
| 20160073055 | Marsh | Mar 2016 | A1 |
| 20160132563 | Bhandari et al. | May 2016 | A1 |
| 20160133297 | Thornton | May 2016 | A1 |
| 20160156992 | Kuper | Jun 2016 | A1 |
| 20160173818 | Su et al. | Jun 2016 | A1 |
| 20160173821 | De Magalhaes | Jun 2016 | A1 |
| 20160198410 | Cherniavsky et al. | Jul 2016 | A1 |
| 20160205716 | Wu et al. | Jul 2016 | A1 |
| 20160212332 | Tang et al. | Jul 2016 | A1 |
| 20160286166 | Sandvik et al. | Sep 2016 | A1 |
| 20160291861 | Song et al. | Oct 2016 | A1 |
| 20160295128 | Schnittman et al. | Oct 2016 | A1 |
| 20160321506 | Fridental | Nov 2016 | A1 |
| 20170004629 | Zhao | Jan 2017 | A1 |
| 20170006162 | Bargetzi et al. | Jan 2017 | A1 |
| 20170041376 | Lin et al. | Feb 2017 | A1 |
| 20170064599 | Caine et al. | Mar 2017 | A1 |
| 20170163748 | Shirriff et al. | Jun 2017 | A1 |
| 20170174068 | Koch et al. | Jun 2017 | A1 |
| 20170244643 | Lawrence et al. | Aug 2017 | A1 |
| 20170324932 | Tangeland | Nov 2017 | A1 |
| 20170353907 | Beattie, Jr. | Dec 2017 | A1 |
| 20180027386 | Zampini, II | Jan 2018 | A1 |
| 20180063482 | Goesnar | Mar 2018 | A1 |
| 20180077592 | Kim et al. | Mar 2018 | A1 |
| 20180091727 | Pell et al. | Mar 2018 | A1 |
| 20180190279 | Anderson et al. | Jul 2018 | A1 |
| 20190028758 | Talvensaari et al. | Jan 2019 | A1 |
| 20190251362 | Meisser et al. | Aug 2019 | A1 |
| 20200077045 | Child et al. | Mar 2020 | A1 |
| 20210051035 | Atkins et al. | Feb 2021 | A1 |
| 20210051036 | Atkins et al. | Feb 2021 | A1 |
| 20210051037 | Atkins et al. | Feb 2021 | A1 |
| 20210051298 | Atkins et al. | Feb 2021 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20220279115 A1 | Sep 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16833854 | Mar 2020 | US |
| Child | 17136832 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17136832 | Dec 2020 | US |
| Child | 17744883 | US |
