Patent Grant
11818419
Mobile devices with graphical user interfaces (GUIs) and wireless communication interfaces are widespread. While permanent storage of content to be display on a mobile device is possible, efforts to expand content provisioning to mobile devices via wireless communication channels are ongoing. There are conflicting limitations related to cameras, display quality, content generation, processing speed, wireless communication channel bandwidth, and system cost that have not been adequately resolved. Efforts to improve content provisioning to HMDs or other mobile devices via wireless communication channels are ongoing.
In accordance with at least one embodiment of the disclosure, a mobile device comprises a display, at least one sensor, and a wireless transceiver. The mobile device also comprises control circuitry coupled to the display, the at least one sensor, and the wireless transceiver. The control circuitry is configured to obtain content primitives from the at least one sensor, to perform content provisioning operations to obtain content based at least in part on the content primitives, and to display the obtained content on the display, wherein at least some of the content is virtual content. In response to a bandwidth condition of the wireless communication channel being less than a threshold, the control circuitry is configured to perform adjusted content provisioning operations that involve increasing an amount of image processing operations performed by the mobile device to obtain the content.
In accordance with at least one embodiment of the disclosure, a method comprises obtaining, by a mobile device, content primitives used to generate content to be displayed by the mobile device. The method also comprises identifying, by the mobile device, a bandwidth condition of a wireless communication channel. In response to the bandwidth condition of the wireless communication channel being less than a threshold, the method comprises increasing, by the mobile device, an amount of image processing operations performed by the mobile device to generate the content. The method also comprises displaying, by the mobile device, the generated content, wherein the generated content includes computer-generated reality (CGR) content.
For a detailed description of various examples, reference will now be made to the accompanying drawings in which:
Disclosed herein are content display devices, systems, and related methods with content provisioning adjustments based on a wireless communication channel bandwidth condition.
In some embodiments, an example content display system includes a mobile device (e.g., a handheld or head-mounted device) and an electronic device (e.g., a desktop computer, laptop computer, a base station or other electronic device) in communication via a wireless communication channel. Over time, the condition of the wireless communication channel bandwidth varies, which affects the amount of data that can be conveyed from the mobile device to the electronic device or vice versa. In response to detected changes in the wireless communication channel bandwidth, content provisioning operations of the mobile device and the electronic device are updated.
In some embodiments, in response to a bandwidth condition of the wireless communication channel being less than a threshold, the content provisioning operations of the mobile device and the electronic device are adjusted by increasing an amount of image processing operations performed by the mobile device and decreasing an amount of content conveyed between the electronic device and the mobile device via the wireless communication channel. Thereafter, in response to a bandwidth condition of the wireless communication channel being greater than a threshold, the content provisioning operations of the mobile device and the electronic device are adjusted by decreasing an amount of image processing operations performed by the mobile device and increasing an amount of content conveyed between the electronic device and the mobile device via the wireless communication channel. Various other content provisioning options and related operations are described herein, where different content provisioning options are used in response to the bandwidth condition of the wireless communication channel (e.g., using one or more bandwidth condition thresholds and/or an amount of time above or below a given bandwidth condition threshold).
Example content provisioning operations performed by the mobile device include, but are not limited to, priority-based transmission of camera images or other content primitives to the electronic device, image/data compression adjustments for images/data to be conveyed to the electronic device via the wireless communication channel, video frames per second adjustments for camera settings and/or captured camera images, image resolution adjustments, foveation filter adjustments, and local data processing adjustments (e.g., changing an amount of image/data processing operations and/or the image/data processing algorithms used). As used herein, “content primitives” refers to any image-based data that can be used to generate content for display. One example of content primitives includes camera image data. In different embodiments, one or more of the content provisioning operations of the mobile device are adjusted based on bandwidth condition thresholds and/or an amount of time above or below a given bandwidth condition threshold
Example content provisioning operations performed by the electronic device include, but are not limited to, rate control adjustments for data to be conveyed to the mobile device, foveation filter adjustments, color space adjustments, image/content frames per second adjustments, and downscaling adjustments (e.g., wavelet decomposition adjustments), content generation (e.g., encoded content/video layers), and providing content (e.g., encoded content/video layers) to a mobile device. In different embodiments, one or more of the content provisioning operations of the electronic device are adjusted based on bandwidth condition thresholds and/or an amount of time above or below a given bandwidth condition threshold. To provide a better understanding, various content display devices, systems, and methods involving content provisioning adjustments performed based on the bandwidth condition of a wireless communication channel are described using the figures as follows.
Example operations of the control circuitry 108 include determining a condition of the wireless communication channel 118, and responding to the determined condition of the wireless communication channel 118 by adjusting content provisioning operations performed by the mobile device 102. In some embodiments, adjustable content provisioning operations of the mobile device 102 include, but are not limited to, priority-based transmission of content primitives (e.g., camera images or other content primitives that can be used to provide content after some processing or other content provisioning operations) to the electronic device 112, image/data compression adjustments for images/data to be conveyed to the electronic device 112 via the wireless communication channel 118, the video frames per second for camera settings and/or captured images, image resolution adjustments, foveation filter adjustments, and local data processing adjustments (e.g., changing an amount of image/data processing operations and/or the image/data processing algorithms used). In different embodiments, one or more of the content provisioning operations of the mobile device 102 are adjusted by the control circuitry 108 based on bandwidth condition thresholds for the wireless communication channel 118 and/or an amount of time above or below a given bandwidth condition threshold for the wireless communication channel 118.
Meanwhile, the electronic device 112 of
Example operations of the control circuitry 114 include determining a condition of the wireless communication channel 118, and responding to the determined condition of the wireless communication channel 118 by adjusting content provisioning operations performed by the electronic device 112. In some embodiments, adjustable content provisioning operations of the electronic device 112 include, but are not limited to, rate control adjustments for data to be conveyed to the mobile device 102, foveation filter adjustments, color space adjustments, image/content frames per second adjustments, and downscaling adjustments (e.g., wavelet decomposition adjustments), content generation (e.g., encoded content/video layers), and providing content (e.g., encoded content/video layers) to the mobile device 102. In different embodiments, one or more of the content provisioning operations of the electronic device 112 are adjusted based on bandwidth condition thresholds for the wireless communication channel 118 and/or an amount of time above or below a given bandwidth condition threshold for the wireless communication channel 118.
In some embodiments, the operations of the control circuitry 108 and the control circuitry 114 vary depending on the mobile device features in use. Example mobile device features may be based on cameras including, but are not limited to, eye cameras, hand tracking cameras, face cameras, depth cameras, and scene cameras. Example mobile device features based on camera images include, but are not limited to, generating content layers, generating 3D objects, video matting, object tracking (e.g., hand-tracking, eye-tracking, face/expression tracking), and/or other features. In different embodiments, the camera settings (e.g., resolution in ppd or pixels/mm, fps, bit rate) may vary for different features or for different modes of the same mobile device feature. When the bandwidth condition of the wireless communication channel 118 is above a threshold, more features and related uses of cameras is possible, where at least some camera image data used is conveyed from the mobile device 102 to the electronic device 112, and related content is conveyed from the electronic device 112 to the mobile device 102. As the bandwidth condition of the wireless communication channel 118 decreases, the set of features or modes being supported can be adjusted, such that the amount of camera image data conveyed from the mobile device 102 to the electronic device 112 via the wireless communication channel 118 is decreased. Similarly, when the set of features or modes being supported is adjusted due to a deteriorated bandwidth condition of the wireless communication channel, the amount of content data conveyed from the electronic device 112 to the mobile device 102 via the wireless communication channel 118 is decreased. In such case, the content provisioning operations of the mobile device 102 and the electronic device 112 are adjusted such that the overall features being supported is temporarily decreased, which reduces the amount of data conveyed via the wireless communication channel 118 (mobile device to electronic device, or electronic device to mobile device).
In one example, the mobile device 102 includes a plurality of sensors 106. In response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve adjusting the operation of the sensors 106, adjusting settings of sensor data captured by the sensors 106, and/or omitting conveyance of at some sensor data captured by the sensors 106 based on a sensor or sensor data prioritization scheme. In some examples, adjusted content provisioning operations of the mobile device 102 involve selectively applying a prioritization scheme to image processing operations to be performed by the mobile device 102 or the electronic device 112 and omitting or migrating some of the image processing operations based on the prioritization scheme.
In one example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, adjusted content provisioning operations of the mobile device 102 or the electronic device 112 involve selectively adjusting a quantity of image data processing algorithms to be used by the mobile device 102 or the electronic device 112.
In one example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, adjusted content provisioning adjustments of the mobile device 102 involve selectively adjusting a compression option for image data captured by the sensor(s) 106 and conveyed from the mobile device 102 to the electronic device 112 via the wireless communication channel 118.
In one example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve selectively adjusting a frames per second for image data captured by the sensor(s) 106 and conveyed from the mobile device 102 to the electronic device 112 via the wireless communication channel 118.
In one example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve selectively adjusting a resolution option for image data by the sensor(s) 106 and conveyed from the mobile device 102 to the electronic device 112 via the wireless communication channel 118.
In one example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve selectively adjusting a foveation option for image data captured by the sensor(s) 106 and conveyed from the mobile device 102 to the electronic device 112 via the wireless communication channel 118.
In one example, the sensor(s) 106 include an eye camera. In response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve the mobile device 102 selectively performing image data processing operations associated with the eye camera. In such case, eye camera data does not need to be conveyed from the mobile device 102 to the electronic device 112, which results in a reduced amount of data being conveyed via the wireless communication channel 118 (mobile device to electronic device, or electronic device to mobile device).
In one example, the sensor(s) 106 include a hand tracking camera. In response to a bandwidth condition of the wireless communication channel being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve the mobile device 102 selectively performing image data processing operations associated with the hand tracking camera. In such case, hand tracking camera data does not need to be conveyed from the mobile device 102 to the electronic device 112, which results in a reduced amount of data being conveyed via the wireless communication channel 118 (mobile device to electronic device, or electronic device to mobile device).
In one example, the sensor(s) 106 include at least one scene camera and at least one face camera. In response to a bandwidth condition of the wireless communication channel being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve the mobile device 102 selectively compressing image data associated with the at least one scene camera more than image data associated with the at least one face camera.
In one example, the sensor(s) 106 include a depth camera, and a scene camera. In response to a bandwidth condition of the wireless communication channel being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve the mobile device 102 selectively adjusting a priority of camera image data associated with the depth camera and the scene camera. In an example prioritization adjustment, some of the data camera image data associated with the depth camera and the scene camera is not conveyed from the mobile device 102 to the electronic device 112 (or is compression more than other types of camera image data), which results in a reduced amount of data being conveyed via the wireless communication channel 118 (mobile device to electronic device, or electronic device to mobile device).
In one example, the sensor(s) 106 include one or more cameras for obtaining head pose tracking data. In response to a bandwidth condition of the wireless communication channel being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve the mobile device 102 selectively processing head pose tracking data and conveying other camera image data to the electronic device 112. In such case, head pose tracking data does not need to be conveyed from the mobile device 102 to the electronic device 112, which results in a reduced amount of data being conveyed via the wireless communication channel 118 (mobile device to electronic device, or electronic device to mobile device).
In one example, the sensor(s) 106 include one or more cameras for obtaining gaze tracking data. In response to a bandwidth condition of the wireless communication channel being less than a threshold, adjusted content provisioning operations of the mobile device 102 involve the mobile device 102 selectively processing gaze tracking data and conveying other camera image data to the electronic device 112. In such case, gaze tracking data does not need to be conveyed from the mobile device 102 to the electronic device 112, which results in a reduced amount of data being conveyed via the wireless communication channel 118 (mobile device to electronic device, or electronic device to mobile device).
In one example, adjusted content provisioning operations of the mobile device 102 involve selectively conveying reduced amounts of captured camera data for matting operations from the mobile device 102 to the electronic device 112 until a threshold channel degradation condition is met, and performing matting operations by the mobile device 102 in response to the threshold channel degradation condition being met. When the mobile device 102 performs matting operations, a reduced amount of data is conveyed via the wireless communication channel 118 (mobile device to electronic device, or electronic device to mobile device).
In one example, adjusted content provisioning operations of the mobile device 102 or the electronic device 112 involve eliminating at least one of 3D room reconstruction operations and object matching operations. By eliminating 3D room reconstruction operations and object matching operations, a reduced amount of data is conveyed via the wireless communication channel 118 (mobile device to electronic device, or electronic device to mobile device).
In one example, the control circuitry 114 for the electronic device 112 and/or the control circuitry 108 for the mobile device 102 is configured to adjust content provisioning operations that affect the content to be displayed on the electronic device 112, where the adjusted content provisioning operations of the electronic device 112 involve different compression options for content data to be conveyed by the transceiver to the mobile device 102 via the wireless communication channel 118. For example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, the control circuitry 108 of the mobile device 102 or the control circuitry 114 for the electronic device 112 adjusts the content provisioning operations of the electronic device 112 by selecting between different channel adaptive rate control options. In another example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, the control circuitry 108 of the mobile device 102 or the control circuitry 114 for the electronic device 112 adjusts content provisioning operations of the electronic device 112 by selecting between different MCS states.
In another example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, the control circuitry 108 of the mobile device 102 or the control circuitry 114 for the electronic device 112 adjusts content provisioning operations of the electronic device 112 by selecting between different foveation filter strength options. In another example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, the control circuitry 108 of the mobile device 102 or the control circuitry 114 for the electronic device 112 adjusts content provisioning operations of the electronic device 112 by selecting between different color space options. In another example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, the control circuitry 108 of the mobile device 102 or the control circuitry 114 for the electronic device 112 adjusts content provisioning operations of the electronic device 112 by selecting between different frame per second options. In another example, in response to a bandwidth condition of the wireless communication channel 118 being less than a threshold, the control circuitry 108 of the mobile device 102 or the control circuitry 114 for the electronic device 112 adjusts content provisioning operations of the electronic device 112 by selecting between different downscaling options.
In some embodiments, the control circuitry 114 for the electronic device 112 selectively combines a plurality of different compression options in response to the bandwidth condition of the wireless communication channel 118 being less than a threshold for a predetermined amount of time, where the plurality of different compression options include data rate options, foveation filter strength options, color space options, frames per second adaptation options, and downscaling options. In one example, the control circuitry 114 for the electronic device 112 adjusts the data rate options as a first priority in response to the bandwidth condition of the wireless communication channel 118 being less than a threshold for a first time interval. Also, the control circuitry 114 for the electronic device 112 adjusts the foveation filter strength options as a second priority in response to the bandwidth condition of the wireless communication channel 118 being less than a threshold for a second time interval greater than the first time interval. Also, the control circuitry 114 for the electronic device 112 adjusts the color space options as a third priority in response to the bandwidth condition of the wireless communication channel 118 being less than a threshold for a third time interval greater than the second time interval. Also, the control circuitry 114 for the electronic device 112 adjusts the frames per second adaptation options as a fourth priority in response to the bandwidth condition of the wireless communication channel 118 being less than a threshold for a fourth time interval greater than the third time interval. Also, the control circuitry 114 for the electronic device 112 adjusts the downscaling options as a fifth priority in response to the bandwidth condition of the wireless communication channel 118 being less than a threshold for a fifth time interval greater than the fourth time interval. Other content provisioning prioritization scheme options, including data compression prioritization options for an electronic device such as the electronic device 112, are possible. Also, dynamic adjustment of content provisioning prioritization scheme options, including data compression prioritization options for an electronic device such as the electronic device 112, are possible. In some examples, dynamic adjustment of content provisioning prioritization scheme options are based on channel condition, user preference, content quality metrics, and/or other criteria. For example, an existing prioritization scheme can be updated to a new prioritization scheme based on the bandwidth condition of a channel, user preference, content quality metrics, and/or other criteria.
As an example, with a maximum MCS state, the mobile device 102 is able to transfer a large amount of content primitives (e.g., sensor data, camera image data and/or other data) to the electronic device 112. Also, with maximum MCS state, the electronic device 112 uses the content primitives to prepare and convey a large amount of content data to the mobile device 102. In such case, the content and/or the mobile device features available for use with the content are enhanced. In contrast, with the minimum MCS state, the mobile device 102 is not able to transfer a large amount of content primitives (e.g., sensor data, camera image data and/or other data) to the electronic device 112. Likewise, with the minimum MCS state, the electronic device 112 is unable to convey a large amount of content data to the mobile device 102. In such case, the mobile device 102 prepares more of the content data, but the content and/or the mobile device features available for use with the content are compromised. Below the minimum MCS state, the mobile device 102 transitions to a stand-alone mode, where the mobile device 102 generates in own content. In such case, the content and/or the mobile device features available for use with the content are compromised. In this example, the mobile device 102 is capable of generating content in a stand-alone mode as needed, but the preferred content display scenario involves an exchange of data between the mobile device 102 and the electronic device 112. Thus, as the MCS state increases, the amount of data exchanged between the mobile device 102 and the electronic device 112 increases along the quality of content and/or mobile device features.
In some examples, mobile device content provisioning options for the baseline distribution of uplink bandwidth of
In some examples, mobile device content provisioning options for the baseline distribution of downlink bandwidth of
In some examples, mobile device content provisioning options for the baseline distribution of uplink bandwidth demand of
In some examples, mobile device content provisioning options for the baseline distribution of downlink bandwidth demand of
As shown, the manager block 702 includes a resource manager 704 that receives voting results from blocks 706, 708, and 710, where block 706 uses active algorithm services to vote, block 708 uses streaming codec stats and rate control states to vote, and block 710 uses channel conditions to vote. The resource manager 704 also receives policy information, user preferences, and/or configuration information from block 716. Based on the votes from blocks 706, 708, and 710, and/or information/preferences from block 716, the response manager 704 provides control outputs to blocks 712 and 714, where block 712 corresponds to a foveation filter and block 714 corresponds to a streaming codec rate control circuitry 714.
The resource manager 704 is configured to communicate with an agent block 720. More specifically, the resource manager 704 is configured to send state sync signals to and to receive state sync signals from a resource agent 722 of the agent block 720. In response to state sync signals from the resource manager 704 and camera compression stats from block 724, the resource agent 722 provides control outputs to a mobile device manager 726 and a camera driver 728. In some embodiments, the foveation filter 712, the streaming codec rate controller 714, the mobile device manager 726, and the camera driver 728 adjust available content provisioning options of a mobile device or electronic device, such as those described for the control circuitry 108 and the control circuitry 114 in
In the example of
In some embodiments, the content provisioning manager 802 corresponds to hardware, firmware, and/or software components of a mobile device such as the mobile device 102 of
In the example of
In the example of
Also, fps information 912 includes instructions, rules, and/or data for selecting fps options in accordance with content provisioning control signals for a mobile device such as the control signals 808. The resolution information 914 includes instructions, rules, and/or data for selecting image/content resolution options in accordance with content provisioning control signals for a mobile device such as the control signals 808. The foveation filter information 916 includes instructions, rules, and/or data for selecting foveation filter options in accordance with content provisioning control signals for a mobile device such as the control signals 808. The algorithm information 918 includes instructions, rules, and/or data for selecting data/image processing algorithm options in accordance with content provisioning control signals for a mobile device such as the control signals 808.
In the example of
In the example of
In one embodiment, the computer system 900 represents components of a mobile device such as the mobile device 102 in
Once control signals (e.g., control signals 808) for content provisioning adjustments are determined (e.g., by the processor(s) 902 executing the control signal information 924), these same control signals or related control signals are used internally within the computer system 900 to perform content provisioning migration. Additionally or alternatively, control signals for content provisioning adjustments are transmitted by the computer system 900 to another device (e.g., electronic device to mobile device, or mobile device to electronic device) by a wireless transceiver 930 in communication with the processor(s) 902.
In the example of
In the example of
In the example of
In some embodiments, the content provisioning manager 1102 corresponds to hardware, firmware, and/or software components of an electronic device such as the electronic device 112 of
In the example of
In the example of
Also, color space information 1212 includes instructions, rules, and/or data for selecting color space options in accordance with content provisioning control signals for an electronic device such as the control signals 1108. The fps adaptation information 1214 includes instructions, rules, and/or data for selecting fps adaptation options in accordance with content provisioning control signals for an electronic device such as the control signals 1108. The downscaling information 1216 includes instructions, rules, and/or data for selecting downscaling options in accordance with content provisioning control signals for an electronic device such as the control signals 1108.
In the example of
In the example of
In one embodiment, the computer system 1200 represents components of an electronic device such as the electronic device 102 in
Once control signals (e.g., control signals 1108) for content provisioning adjustments are determined (e.g., by the processor(s) 1202 executing the control signal information 1224), these same control signals or related control signals are used internally within the computer system 1200 to perform content provisioning migration. Additionally or alternatively, control signals for content provisioning adjustments are transmitted by the computer system 1200 to another device (e.g., electronic device to mobile device, or mobile device to electronic device) by a wireless transceiver 1230 in communication with the processor(s) 1202.
In the example of
In some examples, the content provisioning method 1300 involves mobile device operations at block 1302 including obtaining, by a mobile device, content primitives used to generate content to be displayed by the mobile device. Also, the mobile device operations include identifying, by the mobile device, a bandwidth condition of a wireless communication channel at block 1304. In response to the bandwidth condition of the wireless communication channel being less than a threshold, block 1306 may include the mobile device increasing an amount of image processing operations performed by the mobile device to generate the content. Once the content is generated, the mobile device is able to display the generated content. In some examples, the generated content includes computer-generated reality (CGR) (e.g., augmented reality, mixed reality, virtual reality, and the like) content.
In some examples, the content provisioning method 1300 includes receiving, by an electronic device, content primitives via a wireless communication channel. The method 1300 also includes performing, by the electronic device, content provisioning operations based on the received content primitives, wherein the content provisioning operations involve generating content image data and transmitting the content image data to the wireless communication channel, wherein at least some the content image data corresponds to CGR content. In response to the bandwidth condition of the wireless communication channel being less than a threshold, the method 1300 also includes adjusting the content provisioning operations by decreasing an amount of content image data conveyed to the wireless communication channel.
In one example, in response to the bandwidth condition of the wireless communication channel being less than a threshold, the operations of block 1306 involve adjusting content provisioning operations of the electronic device by decreasing an amount of content data conveyed by the electronic device to the mobile device via the wireless communication channel. Subsequently, if the bandwidth condition of the wireless communication channel improves (e.g., the bandwidth condition is greater than the threshold), the operations of block 1306 involve adjusting content provisioning operations of the electronic device by increasing an amount of content data conveyed by the electronic device to the mobile device via the wireless communication channel.
In some embodiments, the operations of block 1306 involve decreasing or increasing the amount of content data to be conveyed from an electronic device to a mobile device via the wireless communication channel based on different compression options. Example compression options include different channel adaptive rate control options, different MCS states, different foveation filter strength options, different color space options, different frame per second options, and/or different downscaling options. As desired, the use and adjustment of different compression options are selected based on one or more bandwidth condition thresholds such as MCS state, available data rate and/or predetermined time intervals at an MCS state or data rate.
In some embodiments, the operations of block 1306 involve a prioritization scheme to select adjustments. As an example, in some embodiments, adjustments to data rate options, foveation filter strength options, color space options, frames per second adaptation options, and downscaling options are selected based on a predetermined prioritization scheme. In one example, adjusting data rate options is a first priority in response to the bandwidth condition of the wireless communication channel being less than a threshold for a first time interval. Also, adjusting foveation filter rate options is a second priority in response to the bandwidth condition of the wireless communication channel being less than a threshold for a second time interval greater than the first time interval. Also, adjusting the color space options is a third priority in response to the bandwidth condition of the wireless communication channel being less than a threshold for a third time interval greater than the second time interval. Also, frames per second adaptation options is a fourth priority in response to the bandwidth condition of the wireless communication channel being less than a threshold for a fourth time interval greater than the third time interval. Also, adjusting the downscaling options is a fifth priority in response to the bandwidth condition of the wireless communication channel being less than a threshold for a fifth time interval greater than the fourth time interval. In some embodiments, the prioritization scheme is dynamic and adjustable based on user preferences, content quality metrics, and/or other criteria.
In some embodiments, the operations of block 1306 involve decreasing or increasing the amount of content data to be conveyed via the wireless communication channel by increasing an amount of image processing operations performed by the mobile device and decreasing an amount of content conveyed by the electronic device to the mobile device via the wireless communication channel.
Example content provisioning adjustments by a mobile device include, but are not limited to, decreasing an amount of content primitives (e.g., sensor data or camera image data) conveyed from the mobile device to the electronic device, omitting at least some image processing operations performed by at least one of the mobile device and the electronic device, adjusting a quantity of image data processing algorithms to be used by the mobile device or the electronic device, adjusting a compression option for captured image data to be conveyed from the mobile device to the electronic device via the wireless communication channel, adjusting a frames per second for captured image data to be conveyed from the mobile device to the electronic device via the wireless communication channel, adjusting a resolution option for captured image data to be conveyed from the mobile device to the electronic device via the wireless communication channel, and/or adjusting a foveation option for captured image data to be conveyed from the mobile device to the electronic device via the wireless communication channel.
In some embodiments, the operations of block 1306 involve responding to a determination that the wireless communication channel stays less a threshold for multiple time intervals. In response to the bandwidth condition of the wireless communication channel staying less a threshold for multiple time intervals, the amount of content generation operations performed by the mobile device increases for each time interval and the amount of content conveyed by the electronic device to the mobile device decreases for each time interval.
In some embodiments, the operations of block 1306 involve switching image data processing operations associated with an eye camera from the electronic device to the mobile device, switching image data processing operations associated with a hand tracking camera from the electronic device to the mobile device, compressing camera image data associated with the at least one scene camera more than camera image data associated with the at least one face camera, adjusting a priority of camera image data associated with a depth camera and a scene camera of the mobile device, switching processing of head pose tracking data from the electronic device to the mobile device, and/or switching processing of gaze tracking data from the electronic device to the mobile device.
In some embodiments, the operations of block 1306 involve conveying reduced amounts of content primitives (e.g., camera image data) for matting operations from the mobile device to the electronic device until a threshold channel degradation condition is met, and performing matting operations by the mobile device in response to the threshold channel degradation condition being met. In some embodiments, the operations of block 1306 involve eliminating at least some 3D reconstruction operations and/or at least some object matching operations performed by the mobile device or the electronic device.
In some embodiments, the content provision operations of the mobile device and/or the electronic device described herein are employed individually (e.g., only one mobile device adjustment is made, only one electronic device adjustment is made, etc.). In other embodiments, the content provision operations of the mobile device and/or the electronic device described herein are combined in different ways to achieve a desired content quality metric for a given bandwidth condition. For different embodiments, the workload capabilities of the mobile device and/or the electronic device vary, which affects how content provision operations are performed and adjusted in response to the bandwidth condition of a wireless communication channel.
While much of the discussion has been directed to adjusting content provisioning operations of a mobile device and/or electronic device to reduce the amount of content data conveyed via a wireless communication channel in response to a compromised bandwidth condition, it should be appreciated that the bandwidth condition is dynamic. Accordingly, content provisioning adjustments of a mobile device and/or electronic device in response to improvements to the bandwidth condition of a wireless communication channel may also be performed. In some embodiments, control of content provisioning adjustment operations of a mobile device and/or electronic device in response to a dynamic bandwidth condition of a wireless communication channel may be accomplished using a default set of content provisioning operations for each of a plurality of bandwidth conditions. Additionally or alternatively, adjustments to content provisioning operations for different bandwidth conditions of a wireless communication channel is possible based on user preferences, content quality metrics, and/or other criteria.
Certain terms have been used throughout this description and claims to refer to particular system components. As one skilled in the art will appreciate, different parties may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function. In this disclosure and claims, the terms “including” and “comprising” are used in an open-ended fashion, and thus should be interpreted to mean “including, but not limited to . . . .” Also, the term “couple” or “couples” is intended to mean either an indirect or direct wired or wireless connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections. The recitation “based on” is intended to mean “based at least in part on.” Therefore, if X is based on Y, X may be a function of Y and any number of other factors.
The above discussion is meant to be illustrative of the principles and various embodiments of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
The present application claims priority to U.S. Provisional Patent Application No. 62/738,890, filed Sep. 28, 2018.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6970506 | Kim | Nov 2005 | B2 |
| 7076108 | Huang | Jul 2006 | B2 |
| 8471783 | Rhodes | Jun 2013 | B2 |
| 9606363 | Zalewski | Mar 2017 | B2 |
| 20070053428 | Saleem | Mar 2007 | A1 |
| 20080239075 | Mehrotra | Oct 2008 | A1 |
| 20130147973 | Williams | Jun 2013 | A1 |
| 20130195119 | Huang | Aug 2013 | A1 |
| 20130208080 | Lukasik | Aug 2013 | A1 |
| 20160212433 | Zhu | Jul 2016 | A1 |
| 20160241836 | Cole | Aug 2016 | A1 |
| 20170103577 | Mendhekar | Apr 2017 | A1 |
| 20170280166 | Walkingshaw | Sep 2017 | A1 |
| 20170323158 | Gordon | Nov 2017 | A1 |
| 20180033163 | Sugimoto | Feb 2018 | A1 |
| 20180070113 | Phillips | Mar 2018 | A1 |
| 20180132116 | Shekhar | May 2018 | A1 |
| 20180192058 | Chen | Jul 2018 | A1 |
| 20180227187 | Duvanenko | Aug 2018 | A1 |
| 20180288423 | Vembar | Oct 2018 | A1 |
| 20190199467 | Vu | Jun 2019 | A1 |
| 20190356894 | Oh | Nov 2019 | A1 |
| 20200081524 | Schmidt | Mar 2020 | A1 |
| 20200090396 | Holmes | Mar 2020 | A1 |
| 20200288164 | Lin | Sep 2020 | A1 |
| 20200389621 | Ryu | Dec 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 101222296 | Jul 2008 | CN |
| 106408502 | Feb 2017 | CN |
| 2017025483 | Feb 2017 | WO |
| 2017205794 | Nov 2017 | WO |
| 2018106752 | Jun 2018 | WO |
| Entry |
|---|
| Wang, et al., “Foveation Scalable Video Coding with Automatic Fixation Selection,” IEEE Transactions on Image Processing, vol. 12, No. 2, 2003. |
| Ericsson, “CR to CS Multimedia Codec 26.110 for Global Text Telephony in Release 00,” Sep. 8, 2000. |
| Number | Date | Country | |
|---|---|---|---|
| 20200107068 A1 | Apr 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62738890 | Sep 2018 | US |
