Patent Application
20230131669
Brightness settings of an electronic display device allow users to adjust the brightness of a display panel. Some electronic display devices include a single backlight, such that the brightness of the display panel is uniformly modified by brightness adjustments. Some electronic display devices allow for individual pixel dimming or dimming zones or regions, commonly referred to as local dimming. Local dimming allows for dynamic dimming of the display panel backlight based on the displayed content. For example, the backlighting associated with certain zones of pixels, or even individual pixels, may be dimmed when displaying darker scenes and brightened when displaying lighter scenes.
Non-limiting and non-exhaustive examples of the disclosure are described, including various examples of the disclosure, with reference to the figures described below.
According to various examples described herein, a computing device or a display panel controller of the computing device, selectively dims inactive application windows and/or increases the brightness of active application windows. In many instances, a computing device executes multiple applications at the same time. The computing device may display the application windows of several applications on a display panel of the computing device at the same time. However, the user of the computing device may not be actively using all the applications at the same time. For example, a user may leave an email application window open on one portion of the display panel while actively using a word processing application window on another portion of the display panel.
In various examples, the system monitors the gaze direction of the user to determine the user's focus at any given time. At each of a plurality of instances, the system may identify the user focus as being directed to one of the application windows displayed on the display panel of the computing system. The user focus may switch between multiple application windows or may be maintained on a single application window. In some examples, the system identifies application windows as active or inactive and adjusts the relative brightness accordingly.
In various examples, the system adjusts the relative brightness of each application window based on a weighted function of the frequency of the user focus on each respective application, the duration of the user focus on each respective application, and/or the recency of focus on each respective application. In some instances, focus thresholds may be associated with each application. For example, the system may dim the application window of any application for which the frequency, duration, or recency of the user focus is below established threshold values. In some examples, the frequency of focus threshold, the duration of user focus threshold, and the recency of focus threshold of each application are the same. For example, global threshold values can be applied to all applications or to all applications for which application-specific threshold values have not been set by a user, determined via calibration, and/or otherwise established.
In some examples, the system monitors user focus during user-operation of one or more applications during a calibration time period. The system determines application-specific frequency of focus thresholds, duration of user focus thresholds, the recency of focus thresholds, or a combination thereof for each respective application.
In various examples, the system determines the frequency of focus threshold for a given application as the average number of user focus switches to the given application during a defined time unit. For example, a user operating a computing device may be using an email application, a word processing application, and a PDF viewer application. The user may be primarily utilizing the word processing application, but periodically switching user focus to the email application and the PDF viewer application. An example frequency of focus threshold for the email application might be once per 100 seconds (0.01 switches per second), while an example frequency of focus threshold for the PDF viewer application might be once per 1000 seconds (0.001 switches per second). The frequency of focus threshold for the word processing application might be once per 10,000 seconds (0.0001 switches per second).
The system utilizes the frequency of focus threshold to ensure that the application windows of applications that are used frequently (e.g., the email application in the example above) are not dimmed as quickly as the application windows of applications that are used less frequently (e.g., the word processing application). For example, even if the user had taken notes or otherwise utilized the word processing application for a long period of time in the morning, the application window for the word processing application might still be dimmed relatively quickly since the word processing application is considered to be used less frequently than other applications.
The frequency of focus threshold of a given application might vary between different users. For example, a user in a technical role may check their email in the morning, once at lunch, and then again before leaving for the day. The frequency of focus threshold for the email application for the technical user might be once per 25,000 seconds (0.00004 switches per second). In contrast, a user in a salesperson role might check their email every few minutes, and the frequency of focus threshold for the email application for the salesperson might be once per 250 seconds (0.004 switches per second). The system operates to dim the application window as an inverse function of the frequency of focus threshold. Accordingly, the system may operate to dim the application window of the email application for the technical user much more quickly than for the salesperson.
For instance, the system may compute a frequency of focus threshold of 400 seconds for the email application of the salesperson (using an example function in which frequency of focus threshold is computed as 100,000 multiplied by the number of switches per second—100,000×0.004=400). The system may compute the frequency of focus threshold of 4 seconds for the email application of the technical user (again, 100,000×0.00004=4). Using this example function for computing the frequency of focus threshold, the system may dim the application window of the email application of the salesperson if more than 400 seconds have elapsed since the salesperson last focused on the email application. Similarly, the system may dim the application window of the email application of the technical user when more than 4 seconds have elapsed since the technical user last focused on the email application.
In various examples, the system determines the duration of focus threshold for a given application as the average amount of time the user maintains user focus on the given application before switching user focus to a different application. Using the example above, the user may operate the word processing application and the PDF viewer application for extended periods of time, with periodic glances at the email application. Example duration of focus thresholds for the word processing application, the PDF viewer application, and the email application might be 480 seconds, 360 seconds, and 180 seconds, respectively.
As another example, a user may utilize a graphical design application for long periods of time with periodic usage of a conference calling application for quick meetings. The system may dim the application window of the conference calling application relatively quickly as compared to the application window of the graphical design application. The duration of focus threshold for the conference calling application window might be 180 seconds for the particular user. The system may dim the application window of the conference calling application if the last detected user focus occurred more than 180 seconds in the past.
In various examples, the system determines the recency of focus threshold for a given application as an average amount of time that elapses before the user switches the user focus back to the given application (e.g., from a second application back to the application for which the recently of focus threshold is being determined). Continuing with the example above, the user may glance at the email application on a very consistent basis, use the word processing application for extended periods of time, and use the PDF viewer application less frequently. Example recency of focus thresholds might be 20 seconds for the email application, 200 seconds for the word processing application, and 500 seconds from the PDF viewer application.
As an example, a user may utilize a slideshow creation application for long periods of time, but then not return to that application until much later. The recency of focus threshold for the slideshow creation software might be relatively long (e.g., 360 seconds). The system may periodically determine a recency of focus metric for the slideshow creation application. The system may dim the application window of the application window of the slideshow creation application when the recency of focus metric exceeds the recency of focus threshold for the slideshow creation application.
In some examples, the brightness of each application window may be set to a “normal” or “default” brightness, and then dimmed when determined to be inactive. Alternatively, the brightness of each application window may be set to a dim state, and then the brightness may be increased when determined to be active.
In various examples, the system determines the gaze direction of the user via analysis of images captured of the user during operation of the computing device. For example, a camera of the computing device may be used to capture images of the user during operation of the computing device. The system may utilize any of a wide variety of gaze direction detection techniques including, but not limited to, eye tracking, head tracking, focus detection, optical image analysis of corneal reflections, or a combination thereof. In some examples, the camera used to determine the gaze direction may be a full-function optical camera of the computing device useful for other purposes, such as video conferencing, facial recognition, photography, and/or the like.
In some examples, the camera used to determine the gaze direction of the user may be application-specific and used only for determining the gaze direction of the user. Examples of suitable cameras include, but are not limited to, charge-coupled device (CCD)-based cameras, complimentary-metal-oxide semiconductor (CMOS)-based cameras, full color (e.g., RGB) cameras, infrared cameras, near-infrared cameras, and the like. In some examples, the camera may be integrated as a part of the computing device or associated casing. In other examples, the camera may be externally connected to the computing device via a wireless or wired data connection.
In one example, the computing device includes a display panel and a camera to capture images of the user during the use thereof. A gaze detection subsystem may detect gaze directions of the user in the captured images and identify discrete user focuses on a first application window displayed via the display panel. The gaze detection subsystem may be embodied as electronic logic circuitry, a field-programmable gate array, a programmable logic controller, computer-readable instructions executable by a processor, or a combination thereof. The computing device may calculate or compute user focus metrics, such as the frequency of user focus, the duration of user focus, the recency of user focus, or combinations thereof. A brightness control subsystem may modify the relative brightness of the first application window based on computed user focus metrics. The brightness control subsystem may be embodied as electronic logic circuitry, a field-programmable gate array, a programmable logic controller, computer-readable instructions executable by a processor, or a combination thereof.
In some examples, the computing device also includes an application assignment subsystem that determines application-specific focus thresholds, such as a frequency of focus threshold, a duration of focus threshold, a recency of focus threshold, or a combination thereof. The system may compare instantaneous or average user focus metrics of an application with one or more of the focus thresholds to decrease or increase the brightness of the first application window, decrease or increase a brightness of a second application window of a second application executing on the computing device, increase or decrease a brightness a background displayed on the display panel, or a combination thereof.
In some examples, the systems and methods described herein may be implemented by a processor of a computing device executing instructions stored on a non-transitory computer-readable medium. The term non-transitory does not encompass transitory propagating signals. Execution of the instructions stored on a non-transitory computer-readable medium may cause a computing device to perform various operations or functions, as described herein. The executable instructions may be physically or conceptually divided into discrete modules or blocks of instructions, in some examples.
For example, a non-transitory computer-readable medium may include instructions stored thereon that, when executed by a processor of a computing device, cause the computing device to detect gaze directions of a user captured images, compute a discrete user focus for each captured image on a first application window associated with a first application, and then adjust the relative brightness of the first application based on user focus metrics. Again, the user focus metric may include a frequency of user focus on the first application window, a duration of user focus on the application window, a recency of focus on the first application window, or a combination thereof. Adjustment of the relative brightness of the application window may be based on a comparison of one or more of the user focus metrics with user focus thresholds, such as a frequency of focus threshold, a duration of focus threshold, a recency of focus threshold, or a combination thereof.
As used herein, a basic input/output system (BIOS) refers to hardware or hardware and instructions to initialize, control, or operate a computing device prior to execution of an operating system (OS) of the computing device. Instructions included within a BIOS may be software, firmware, microcode, or other programming that defines or controls functionality or operation of a BIOS. In one example, a BIOS may be implemented using instructions, such as platform firmware of a computing device, executable by a processor. A BIOS may operate or execute prior to the execution of the OS of a computing device. A BIOS may initialize, control, or operate components such as hardware components of a computing device and may load or boot the OS of computing device.
In some examples, a BIOS may provide or establish an interface between hardware devices or platform firmware of the computing device and an OS of the computing device, via which the OS of the computing device may control or operate hardware devices or platform firmware of the computing device. In some examples, a BIOS may implement the Unified Extensible Firmware Interface (UEFI) specification or another specification or standard for initializing, controlling, or operating a computing device.
The computing device 200 also includes a camera 230, such as an infrared camera, a near-infrared camera, a visible light optical camera, a lidar imaging system, or a combination thereof. The computing device 200 uses the camera 230 to capture images of a user during operation of the computing device 200. For example, the camera 230 may capture an image or sequence of images of the user during the operation of an application executing on the computing device 200, where the application includes an application widow displayed via the display panel 220.
The computing device 200 includes a gaze detection subsystem 240 that may, for example, be embodied as executable instructions, a field-programmable gate array, a programmable logic device, or a combination thereof. The gaze detection subsystem 240 operates to analyze the captured images to detect gaze directions of the user in the captured images. The gaze detection subsystem 240 maps the detected gaze directions to the surface of the display panel 220 to identify a discrete user focus on an application window displayed thereon. The computing device 200 includes a brightness control subsystem 250 that, again, may be embodied as executable instructions, a field-programmable gate array, a programmable logic device, or a combination thereof. The brightness control subsystem 250 operates to modify the relative brightness of an application window based on a function (e.g., a weighted function) of user focus metrics. For example, the brightness of an application window may be adjusted based on the frequency of user focus, a duration of the user focus, a recency of the user focus, or a combination thereof (including a weighted or ordered combination thereof).
As illustrated, the computer-readable storage medium 350 includes a gaze detection module 351, a user focus detection module 352, a user focus metric calculation module 353, and a brightness control module 354. The gaze detection module 351 may, for example, include instructions that, when executed by the processor 310, cause the computing system 300 to analyze captured images to detect gaze directions of the user. The user focus detection module 352 maps detected gaze directions to the surface of the display panel 340 to identify discrete user focuses on an application window displayed on the display panel 340.
The user focus metric calculation module 353 includes instructions that, when executed by the processor, cause the computing system 300 to identify applications as either active or inactive based on a weighted function of the frequency of the user focus on each respective application, the duration of the user focus on each respective application, the recency of focus on each respective application, or a combination thereof. The brightness control module 354 includes instructions that, when executed by the processor, cause the computing system 300 to adjust the relative brightness of portions of a display panel 340 of the computing system 300. For example, the computing device may operate to dim applications identified as inactive, increase the brightness of applications identified as active, or a combination thereof.
Each application operating on the system may be associated with user focus thresholds as described herein. If a determined duration of focus metric for a given application fails to satisfy the duration of focus threshold, at 415, the application is identified as inactive, at 410. Similarly, if a determined frequency of focus metric for the application fails to satisfy the frequency of focus threshold, at 420, the application is identified as inactive, at 410. Likewise, if a determined recency of focus threshold metric for the application fails to satisfy the recency of focus threshold, at 425, the application is identified as inactive 410.
If the determined user focus metrics satisfy the established user focus thresholds, at 415, 420, and 425, then the application is identified as active 427. The method may be completed for each application in the application window, at 430. The system may implement the computation of the user focus metrics, at 407, and comparisons with the user focus thresholds, at 415, 420, and 425 for multiple applications at the same time or sequentially. The system dims portions of the display that are not associated with active applications windows, at 435. In some examples, the system dims portions of the display that are associated with inactive application windows. In some examples, the system increases the brightness of portions of the display that are associated with active application windows.
The frequency of focus user metrics is expressed as a number of switches in user focus from another (different) application window to the first application window 510. The recency of user focus metrics is expressed as a time since the user last switched user focus between the first application window 510 and another (different) application window. The duration of user focus metric is expressed as an average duration of user focus on the first application window 510 (e.g., the average during a recent time period, such as the last few minutes, hour(s), or day(s)).
Based on the example user focus metrics in table 680, the user looks at (e.g., focuses on) the first application window 610 once every 100 seconds for an average of 100 seconds each time. The user focuses on the second application window 620 every 1000 seconds for an average of 1 second each time. It has been 200 seconds since the user last focused on the second application window 620. In the illustrated example, the system has identified the first application window 610 as active and the second application window 620 as inactive. The background application window 690 is also identified as inactive.
Based on the example user focus metrics in table 780, the user looks at (e.g., focuses on) the first application window 710 once every 100 seconds for an average of five seconds each time. The user last looked at the first application window 710 10 seconds ago. The user focuses on the second application window 720 every 100 seconds for an average of 5 seconds each time. It has been 5 seconds since the user last focused on the second application window 720. In the illustrated example, the system has identified the first application window 710 as active, the second application window 720 as active, and the background application window 790 as inactive.
For instance, the background application window 890 may be dimmed when any number of other application windows are identified as active. Similarly, the system may increase the brightness of the background application window 890 when all other displayed application windows are identified as inactive. In alternative examples, the background application window may be treated similarly to the other application windows and have its brightness increased and decreased based on computed user focus metrics.
Based on the example user focus metrics in table 880, the user looks at (e.g., focuses on) the first application window 810 once every 2 seconds for an average of 2 seconds each time. The user last looked at the first application window 810 10 seconds ago. Based on the relatively high frequency of user focus, the relatively low recency of focus, the relatively low duration of focus, or a combination thereof, the system determines that the first application window 810 is inactive. Since there are no other application windows displayed on the display panel 800, the system increases the brightness of the background application window 890.
Specific examples of the disclosure are described above and illustrated in the figures. It is, however, appreciated that many adaptations and modifications can be made to the specific configurations and components detailed above. In some cases, well-known features, structures, and/or operations are not shown or described in detail. Furthermore, the described features, structures, or operations may be combined in any suitable manner. Thus, all feasible permutations and combinations of examples are contemplated.
In the description above, various features are sometimes grouped together in a single example, figure, or description thereof for the purpose of streamlining the disclosure. This method of disclosure, however, is not to be interpreted as reflecting an intention that any claim now presented or presented in the future requires more features than those expressly recited in that claim. Rather, it is appreciated that inventive aspects lie in a combination of fewer than all features of any single foregoing disclosed example. The claims are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate example. This disclosure includes all permutations and combinations of the independent claims with their dependent claims.
