Patent Application
20240274100
The present invention relates to frame rate control method, and, in particular, to frame rate control method in multi-window scenario.
Accompanying the rapid development of electronic products and fast changes in communication technology, mobile devices such as smartphones and tablets have become indispensable. Along with the popularization of these technologies, many consumers have transferred their regular daily online activity from a PC platform to their mobile devices, such as browsing webpages, receiving and sending emails, and viewing media content.
Currently, a multi-window display technology is being used widely on mobile devices, especially foldable phones. In a multi-window scenario, a plurality of windows are displayed on the screen of said mobile device. Therefore, multi-window resource control of the mobile device is desired for a better user experience.
An embodiment of the present invention provides a frame rate control method. A primary scenario and a non-primary scenario are identified according to two or more windows displayed on a screen. Each of the primary scenario and the non-primary scenario is performed by an individual application. The frame rate of the non-primary scenario is decreased when the performance index indicates that the first condition is present (or met). The application corresponding to the non-primary scenario is disabled when the performance index indicates that the second condition is met after decreasing the frame rate of the non-primary scenario, so as to remove the window corresponding to the non-primary scenario from the screen.
Furthermore, an embodiment of the present invention provides a frame rate control method. A primary scenario and a non-primary scenario are identified in the multi-window scenario of a mobile device. Each of the primary scenario and the non-primary scenario is displayed in an individual window on the screen of the mobile device. The frame rate of the non-primary scenario is decreased and the frame rate of the primary scenario is kept when the performance index indicates that the first condition is present. The frame rate of the non-primary scenario is decreased to the first frame rate and the frame rate of the primary scenario is decreased to the second frame rate when the performance index indicates that the second condition is present. The first frame rate is lower than the second frame rate.
Moreover, an embodiment of the present invention provides a controller for controlling frame rates in multi-window scenario of a mobile device. A processor is configured to identify a primary scenario and a non-primary scenario according to two or more windows displayed on a screen of the mobile device. The processor is coupled to the screen. When a temperature of the mobile device is equal to or greater than a first threshold value, the processor is configured to decrease the frame rate of the non-primary scenario. When the temperature of the mobile device is higher than a second threshold value (which is, itself, higher than the first threshold value), the processor is configured to close the non-primary scenario, so as to remove the window corresponding to the non-primary scenario from the screen.
The present invention can be more fully understood by reading the subsequent detailed description and examples with references made to the accompanying drawings, wherein:
The following description is of the best-contemplated mode of carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims.
Some variations of the embodiments are described. Throughout the various views and illustrative embodiments, like reference numbers are used to designate like elements. It should be understood that additional operations can be provided before, during, and/or after a disclosed method, and some of the operations described can be replaced or eliminated for other embodiments of the method.
The controller 110 includes at least one processor 112 and at least one memory 114. The processor 112 may include one or more processing units. For example, the processor 112 may include a central processing unit (CPU), an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural-network processing unit (neural-network processing unit, NPU). Different processing units may be independent components, or may be integrated into one or more processors. The memory 114 is configured to store instructions and data. The memory 114 may store instructions or data that is used by the processor 112. By using the processor 112 and the memory 114, the controller 110 may be a nerve center and a command center of the mobile device 100. The controller 110 may perform the applications according to instruction operation code and a time sequence signal.
The thermal sensor 140 is configured to sense a temperature of the mobile device 100. In some embodiments, the thermal sensor 140 is an independent sensor disposed on a circuit board, and the thermal sensor 140 is located near thermal source(s) such as the controller 110, the display module 120 and/or the battery 130. In some embodiments, the thermal sensor 140 may be an on-chip thermal sensor used for sensing a junction temperature, a skin temperature, and/or a printed circuit board (pcb) temperature of the controller 110.
The display module 120 includes the screen 122 and the interface unit 124. The screen 122 is configured to display the image, video or the like according to the frame data from the controller 110. The screen 122 is implemented in a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-OLED, quantum dot light-emitting diodes (QLED), or the like. The interface unit 124 is configured to provide the user input to the controller 110. In some embodiments, the screen 122 and the interface unit 124 are implemented in a touch panel.
In a multi-window scenario, multiple windows are displayed on the screen 122. The multi-window scenario may specifically include a plurality of application scenarios, for example, a multi-application multi-window scenario, a single-application multi-window scenario, a split-screen multi-application multi-window scenario, or a picture-in-picture scenario. For example, in the multi-application multi-window scenario, multiple application windows are displayed on the screen 122, and a size and a location of each window may be arbitrary. The size and the location of the window can be modified by dragging the window by the user through the interface unit 124. In some embodiments, different windows may be displayed in a cascade manner. In some embodiments, the windows displayed on the screen 122 may occupy the entire screen 122. In some embodiments, the windows displayed on the screen 122 may only occupy a part of the screen 122.
When multiple windows are displayed on the screen 122 in the multi-window scenario, the controller 110 is capable of detecting the scenario of each window to regulate the operations of the scenarios, so as to improve user experience for the primary scenario under the limited resources of the mobile device 100.
First, in step S210, according to the applications performed by the processor 112, the controller 110 is configured to detect that multi-window scenario is present, i.e., multiple windows are displayed on the screen 122. Furthermore, each window is configured to provide (or display) a scenario performed by an individual application. In other words, the processor 112 is configured to simultaneously perform the various applications for the user to play game, watch video or Live Broadcast, or use video call.
In step S220, the controller 110 is configured to identify a primary scenario and a non-primary scenario (or more non-primary scenarios) in the multi-window scenario, and the primary scenario and the non-primary scenario are provided (displayed) by different windows on the screen 122. In some embodiments, the primary scenario and the non-primary scenario are identified according to the user behavior index, and the user behavior index includes information regarding the behavior of the user operating the screen 122, e.g., the user input from the interface unit 124. In some embodiments, the user behavior index indicates which window on the screen 122 was viewed or focused. In some embodiments, the primary scenario is identified according to the user behavior index indicating which window was last touched or which window was touched the most times in a specific time. In some embodiments, the primary scenario is identified according to the user behavior index indicating which window has the longest usage time. In some embodiments, the primary scenario is identified according to the user behavior index indicating which window has the largest window size. In some embodiments, the primary scenario and the non-primary scenario are identified by AI (artificial intelligence). In some embodiments, the primary scenario and the non-primary scenario are determined by the user predetermined setting, e.g., the importance of each scene is assigned by the user.
Referring to
In some embodiments, the primary scenario is identified according to the user behavior index that is selected (assigned) to indicate which window was last touched. For example, the window 310 was last touched by the user, as shown in label 315 in
In some embodiments, the primary scenario is identified according to the user behavior index that is selected (assigned) to indicate the window having the most touches. For example, in a specific time period (e.g., the last five minutes), the window 310 was touched 3 times, and the second window was touched 12 times, i.e., the number of touches of the window 310 is lower than the number of touches of the window 320 during the last five minutes. According to the user behavior index, the controller 110 is configured to determine (or identify) that the first scenario is the non-primary scenario since the window 310 has the least number of touches during the last five minutes, and the second scenario is the non-primary scenario since the window 320 has the most touches during the last five minutes.
In some embodiments, the primary scenario is identified according to the user behavior index that is selected (assigned) to indicate the window focused by the user. For example, the user behavior index indicates which one of the windows 310 and 320 is viewed by the user, which audio has been played in the window 310 or 320 through a speaker of the mobile device 100 or which one of the windows 310 and 320 has larger sound level.
Referring to
In
Referring back to step S230 of
In step S240, according to the performance index, the controller 110 is configured to determine whether to re-allocate resource for maintaining or optimizing the performance of the primary scenario. If there is no need to re-allocate resources (i.e., no frame dropping or frame jitter and the remaining battery power and the sensed temperature are normal), the flowchart is returned to step S230, and the performance of the primary scenario is monitored again. If resources need to be re-allocated, the controller 110 is configured to obtain (or calculate) the target performance (e.g., the target frame rate) of the primary scenario (step S250).
In step S260, the controller 110 is configured to adjust the resources of the mobile device 100 for the primary scenario, and then the flowchart is returned to step S230. Thus, the performance of the primary scenario is monitored again after adjusting the resources of the mobile device 100.
In some embodiments, when the performance index indicates that the frame dropping or frame jitter is present in the primary scenario (step S240), the controller 110 is configured to reduce the resource of the non-primary scenario (step S260) so as to preserve the frame rate of the primary scenario or improve the performance of the primary scenario (e.g., the target performance obtained in step S250). In some embodiments, reducing the resource of the non-primary scenario includes decreasing the frame rate of the non-primary scenario, assigning Android package (APK) to certain CPU, or decreasing the resolution of the non-primary scenario. In general, the frame dropping (e.g., some frames may be dropped before reaching the display module 120) or the frame jitter (e.g., the frame rate is unstable) is caused by the overloaded processor 112. The frame dropping or the frame jitter often causes graphics contents to stutter and pause on the screen, which can negatively impact the user experience. In some embodiments, after the resource (such as the frame rate or the resolution) of the non-primary scenario is decreased and the frame dropping or frame jitter is still present in the primary scenario, the controller 110 is configured to disable the non-primary scenario (i.e., only the window corresponding to the primary scenario is displayed on the screen 122), so as to preserve the frame rate of the primary scenario or improve the performance of the primary scenario.
In some embodiments, when the performance index indicates that the component state is bad (step S240), the controller 110 is configured to reduce the resource of the non-primary scenario (step S260) so as to preserve the frame rate of the primary scenario or improve the performance of the primary scenario (e.g., the target performance obtained in step S250). In some embodiments, reducing the resource of the non-primary scenario includes decreasing the frame rate of the non-primary scenario, assigning Android package (APK) to certain CPU, or decreasing the resolution of the non-primary scenario. The component state is deemed to be bad when certain conditions are met, such as when the remaining battery power of the battery 130 is lower than the battery threshold value, or when the temperature sensed by the thermal sensor 140 is higher than the thermal threshold value of the mobile device 100. In some embodiments, if the component state is still bad after the resource (such as the frame rate or the resolution) of the non-primary scenario is decreased, the controller 110 is configured to disable the non-primary scenario (i.e., only the window corresponding to the primary scenario is displayed on the screen 122), so as to preserve the frame rate of the primary scenario or improve the performance of the primary scenario.
In some embodiments, the resource of the non-primary scenario may be decreased (steps S250, S260) directly after step S220. That is, the resource of the non-primary scenario is decreased without further determination (e.g. steps S230, S240 are skipped) after the primary scenario and non-primary scenario are identified (step S220).
The APP process module 510 is configured to perform a first application for display a first scenario in a first window on the screen 122. The APP process module 510 includes a blast buffer queue (BBQ) 512, and the frames of the first scenario are displayed based on the BBQ 512 and related settings. The APP process module 520 is configured to perform a second application for displaying a second scenario in a second window on the screen 122. The APP process module 520 includes a BBQ 522, and the frames of the second scenario are displayed based on the BBQ 522 and related settings.
The WMS module 530 is configured to notify window priority of the first and second windows on the screen 122. Furthermore, the WMS module 530 is further configured to provide the foreground applications list that recording the first application corresponding to the first scenario and the second application corresponding to the second scenario. In some embodiments, the user behavior index is provided by the WMS module 530.
The resource allocation module 540 includes the frame rate operation service (FOS) 542 and the frame rate smoother (FRS) 544. The FOS 542 is configured to detect whether multi-window scenario is present and to identify the primary scenario and the non-primary scenario in the multi-window scenario according to the information from the WMS module 530.
The FRS 544 is configured to obtain the performance index with the information regarding the frame rate of each scenario and the component state of the mobile device 100. In response to the performance index and the default frame rates of the various scenarios, the FRS 544 is configured to determine whether to re-allocate resources for the primary scenario in multi-window scenario. In some embodiments, the FRS 544 is configured to adjust the resources of the mobile device 100 to meet the target performance (e.g., the target frame rate) of the first and second scenarios. Furthermore, the resource allocation module 540 is further configured to provide the target performance to the FPSGO 550.
The FPSGO 550 is configured to provide the new frame rates and related settings corresponding to the target performance to the resource allocation module 540. In response to the new frame rates and related settings, the resource allocation module 540 is configured to provide the frame rate controlling settings to the APP process modules 510 and 520 for the first and second scenarios. Furthermore, FPSGO 550 is further configured to provide the new frame rates and related settings corresponding to the target performance to the main process module 560, so that the tasks performed in the CPU 561 and/or GPU 563 can be adjusted accordingly.
In the embodiment of
When the performance index indicates that the sensed temperature is higher than or equal to 45° C. after decreasing the frame rate of the non-primary scenario to 60 FPS, the resource allocation module 540 is configured to decrease the frame rate of the non-primary scenario from 60 FPS to 45 FPS. Simultaneously, the frame rate of the primary scenario is dynamically controlled by the FRS 544 of the resource allocation module 540, such as the frame rate of the primary scenario may be kept at the default frame rate or decreased to 80% of the default frame rate. In some embodiments, when the sensed temperature drops below 45° C. after decreasing the frame rate of the non-primary scenario to 45 FPS, the frame rate of the non-primary scenario is increased to 60 FPS.
When the performance index indicates that the sensed temperature is higher than or equal to 47° C. after decreasing the frame rate of the non-primary scenario to 45 FPS, the resource allocation module 540 is configured to decrease the frame rate of the non-primary scenario from 45 FPS to 30 FPS. Simultaneously, the frame rate of the primary scenario is dynamically controlled by the FRS 544 of the resource allocation module 540, such as the frame rate of the primary scenario may be kept at the default frame rate or decreased to 80% of the default frame rate. In some embodiments, when the sensed temperature drops below 47° C. after decreasing the frame rate of the non-primary scenario to 30 FPS, the frame rate of the non-primary scenario is increased to 45 FPS.
When the performance index indicates that the sensed temperature is higher than or equal to 50° C. after decreasing the frame rate of the non-primary scenario to 30 FPS, the resource allocation module 540 is configured to close (or remove) the non-primary window, i.e., the non-primary scenario is disabled. Simultaneously, the frame rate of the primary scenario is dynamically controlled by the FRS 544 of the resource allocation module 540, such as the frame rate of the primary scenario may be kept at the default frame rate or decreased to 80% of the default frame rate. In other words, the controller 110 is configured to stop performing the application of the non-primary scenario (i.e., the multi-window scenario is absent), so as to reserve resources for the primary scenario.
In the embodiment of
When the performance index indicates that the remaining battery power of the battery 130 is between 31% and 40% after decreasing the frame rate of the non-primary scenario to 60 FPS, the resource allocation module 540 is configured to decrease the frame rate of the non-primary scenario from 60 FPS to 45 FPS. In some embodiments, when the battery 130 is charged and the remaining battery power of the battery 130 is higher than 40% after decreasing the frame rate of the non-primary scenario to 45 FPS, the frame rate of the non-primary scenario is increased to 60 FPS.
When the performance index indicates that the remaining battery power of the battery 130 is between 20% and 30% after decreasing the frame rate of the non-primary scenario to 45 FPS, the resource allocation module 540 is configured to decrease the frame rate of the non-primary scenario from 45 FPS to 30 FPS. In some embodiments, when the battery 130 is charged and the remaining battery power of the battery 130 is higher than 30% after decreasing the frame rate of the non-primary scenario to 30 FPS, the frame rate of the non-primary scenario is increased to 45 FPS.
When the performance index indicates that the remaining battery power of the battery 130 is lower than 20% after decreasing the frame rate of the non-primary scenario to 30 FPS, the resource allocation module 540 is configured to close (remove) the non-primary window, i.e., the non-primary scenario is disabled. In other words, the controller 110 is configured to stop performing the application of the non-primary scenario (i.e., the multi-window scenario is absent), so as to reserve resources for the primary scenario.
Embodiments for controlling frame rates in multi-window scenario of the mobile device 100 are provided. The primary scenario is identified from the windows on the screen 122. The performance of the primary scenario and the operating status of the mobile device 100 are monitored, so as to dynamically adjust resources for the primary scenario. Compared with the conventional multi-window scenario that cannot adjust the frame rate of each scenario, the frame rate of non-primary scenario is dynamically decreased to provide the resource for the primary scenario in the mobile device 100, so as to obtain higher average and standard deviation of the frame rate in the primary scenario, thereby optimizing the user experience. Furthermore, the power consumption of the mobile device 100 is also decreased due to the decreased frame rate of the non-primary scenario or the disabled non-primary scenario.
While the invention has been described by way of example and in terms of the preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements (as would be apparent to those skilled in the art). Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202311826091.X | Dec 2023 | CN | national |
This application claims the benefit of U.S. Provisional Application No. 63/484,759, filed Feb. 14, 2023, and China patent application Ser. No. 202311826091.X filed on Dec. 16, 2023, the entirety of which are incorporated by reference herein.
| Number | Date | Country | |
|---|---|---|---|
| 63484759 | Feb 2023 | US |
