Patent Application
20250217928
This application claims the priority benefit of Taiwan application serial no. 112151708, filed on Dec. 29, 2023. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification.
The disclosure relates to an image resolution adjustment method and an electronic device.
The resolutions and the frame rates of image capturing that can be supported by the camera modules of most smartphones, tablets, or laptops have gradually increased with the advancement of technology. However, for most camera modules, if the resolution of the captured image is increased, the frame rate of the captured image will be inevitably decreased accordingly. On the contrary, if the frame rate of the captured image is increased, the resolution of the captured image will be inevitably decreased accordingly. Therefore, in a situation that a slow motion photographing mode supported by some types of camera modules is switched, since the frame rate of the captured image is increased, the resolution of the captured image will also be correspondingly decreased. Later, when viewing the image previously captured through the slow motion photographing mode on a larger screen, the user will notice a significant decrease in image quality.
The disclosure provides an image resolution adjustment method and an electronic device, which can improve the aforementioned problems.
An embodiment of the disclosure provides an image resolution adjustment method which includes: executing a target application to obtain a target image with a first resolution based on a first frame rate; during an execution of the target application, selecting a target memory from multiple buffer memories according to a usage rate of each of the buffer memories; processing the target image by an image processing model and the target memory to adjust a resolution of the target image from the first resolution to a second resolution, where the second resolution is higher than the first resolution; and controlling a display interface to present the target image with the second resolution based on the first frame rate.
An embodiment of the disclosure further provides an electronic device which includes an image capturing interface, a display interface, multiple buffer memories, and a processor. The processor is coupled to the image capturing interface, the display interface, and the buffer memories. The processor is configured to: execute a target application to instruct the image capturing interface to obtain the target image with a first resolution based on a first frame rate; select a target memory from the buffer memories according to a usage rate of each of the buffer memories during an execution of the target application; process the target image by an image processing model and the target memory to adjust a resolution of the target image from the first resolution to a second resolution, where the second resolution is higher than the first resolution; and control the display interface to present the target image with the second resolution based on the first frame rate.
Based on the above, the target image with the first resolution may be obtained by executing the target application based on the first frame rate. On the other hand, during the execution of the target application, the target memory may be selected from the buffer memories according to the usage rate of each of the buffer memories. The target image may be processed by the image processing model and the target memory to increase the resolution of the target image from the first resolution to the second resolution. Thereafter, the display interface may be controlled to present the target image with the second resolution based on the first frame rate. In this way, in a premise that the image processing performance of the image processing model may be improved as much as possible, the requirement of the user for photographing and presenting the image with a high resolution based on a higher frame rate may be met.
The electronic device 10 includes an image capturing interface 11, a display interface 12, a storage circuit 13, buffer memories 14(1)˜14(n), and a processor 15. The image capturing interface 11 is configured to capture an external image. For example, the image capturing interface 11 may include an image capturing module. This image capturing module includes at least a lens and a photosensitive element. This image capturing module (including the lens and the photosensitive element) may be configured to photograph the external image and generate corresponding image data.
The display interface 12 is configured to display an image. For example, the display interface 12 may include a plasma display, a liquid-crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic light-emitting diode (OLED), and a light-emitting diode display (LED display), and a type of the display interface 12 is not limited thereto.
The storage circuit 13 is configured to store data. For example, the storage circuit 13 may include a read only memory (ROM), a solid state disk (SSD), a conventional hard disk drive (HDD), a flash memory module, an embedded multimedia card (eMMC), a universal flash storage (UFS) device, or other types of non-volatile storage media.
The buffer memories 14(1)˜14(n) are configured to store data temporarily. For example, the buffer memories 14(1)˜14(n) may include a dynamic random access memory (DRAM) or other types of volatile storage media. In addition, the number of buffer memories 14(1)˜14(n) is multiple (that is, n is greater than 1).
The processor 15 is coupled to the image capturing interface 11, the display interface 12, the storage circuit 13, and the buffer memories 14(1)˜14(n). The processor 15 may be responsible for the whole or part of the operation of the electronic device 10. For example, the processor 15 may include a central processing unit (CPU) or other programmable microprocessors for a common purpose or a specific purpose, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a programmable logic device (PLD), other similar devices, or a combination of these devices. In an embodiment, the processor 15 may also include a graphic processing unit (GPU), a vision processing unit (VPU), a neural network processing unit (NPU), or other processors dedicated to (or facilitating to) perform image processing or neural network processing.
In an embodiment, an application 101 (also referred to as a target application) may be stored in the storage circuit 13. The application 101 is configured to control the image capturing interface 11. For example, the processor 15 may perform the operations on the image capturing interface 11, such as setting a parameter, selecting an operating mode, and/or starting the image capturing interface 11 to capture the external image by the application 101. For example, the application 101 may include a camera application or other types of applications configured to control the image capturing interface 11.
In an embodiment, an image processing model 102 may further be stored in the storage circuit 13. The image processing model 102 may include a machine learning model, a deep learning model, and/or other computer computing engines. In addition, the image processing model 102 may adopt a neural network such as a deep neural network (DNN), a recurrent neural network (RNN), and/or a convolutional neural network (CNN), or various computing architectures such as an artificial neural network (ANN), which is not limited by the disclosure.
In an embodiment, the processor 15 may execute the application 101 to obtain an image (also referred to as a target image) with a specific resolution (also referred to as a first resolution) based on a certain frame rate (also referred to as a first frame rate). For example, after starting the application 101, the processor 15 may control the image capturing interface 11 by the application 101. For example, the processor 15 may instruct the image capturing interface 11 to capture (for example, photograph) the target image by the application 101 based on the first frame rate and the first resolution.
In an embodiment, the first frame rate may be 120 frames per second (FPS), and the first resolution may be 640×360. In an embodiment, the first frame rate needs to be at least 120 FPS. In an embodiment, the first frame rate may be at or close to a highest frame rate supported by the image capturing interface 11. In an embodiment, the first frame rate and first resolution may also be adjusted according to practical requirements.
In an embodiment, the processor 15 may control the image capturing interface 11 by the application 101 to operate in a specific photographing mode to capture (for example, photograph) the target image based on the first frame rate and the first resolution. For example, this specific photographing mode may include a slow motion photographing (or videoing) mode.
In an embodiment, due to the hardware limitation of the image capturing interface 11, in the specific photographing mode (for example, the slow motion photographing (or videoing) mode), the image capturing interface 11 captures the target image based on the first frame rate, so the resolution of the target image (that is, the first resolution) may not reach (or may even be much lower than) another resolution (also referred to as a second resolution) supported by the image capturing interface 11. In an embodiment, the second resolution needs to at least reach a requirement of full high definition (FHD). For example, the second resolution may be at least 1920×1080 or 2560×1440. In an embodiment, the second resolution may be at or close to the highest resolution supported by the image capturing interface 11. In an embodiment, the second resolution may also be adjusted according to practical requirements.
In an embodiment, during an execution of the application 101, the processor 15 may monitor a usage rate of each of the buffer memories 14(1)˜14(n). The processor 15 may select at least one memory (also referred to as a target memory) from the buffer memories 14(1)˜14(n) according to the monitored usage rate of each of the buffer memories 14(1)˜14(n). For example, the usage rate of each of the buffer memory in the buffer memories 14(1)˜14(n) may reflect (for example, be positively related to) the frequency with which the buffer memory is used or accessed. For example, during the execution of the application 101, if the usage rate of a certain buffer memory in the buffer memories 14(1)˜14(n) is relatively high, it means that the buffer memory is more often (for example, more frequently) used or accessed than other buffer memories.
In an embodiment, the processor 15 may compare the usage rate of each of the buffer memories 14(1)˜14(n) during the execution of the application 101. Then, processor 15 may select at least one buffer memory from the buffer memories 14(1)˜14(n) as the target memory according to the comparison result. In an embodiment, the target memory may be the buffer memory with the highest or relatively high usage rate in the buffer memories 14(1)˜14(n) during the execution of the application 101.
In an embodiment, after selecting the target memory, the processor 15 may process the target image by the image processing model 102 and the target memory to adjust (for example, increase) the resolution of the target image from the first resolution to the second resolution. In particular, the second resolution may be higher than the first resolution. For example, assuming that the first resolution is 640×360, the second resolution may be 1920×1080 or 2560×1440, and the disclosure is not limited thereto.
In an embodiment, in a process of processing the target image, the image processing model 102 may increase a resolution of each image frame in the target image by four times (or other multiples) to increase the resolution of the target image from the first resolution to the second resolution. In an embodiment, in the process of processing the target image, the image processing model 102 may further perform various image processing methods on the target image with improved resolution, such as pixel value interpolation and/or edge smoothing to improve the image quality of the target image with the improved resolution.
In an embodiment, after selecting the target memory, the processor 15 may dispose one or multiple reserved spaces in the target memory. This reserved space may be dedicated to storing data used by the image processing model 102 in the process of processing the target image. For example, the processor 15 may dispose this reserved space to be dedicated to temporarily storing various (transient) data, such as tensor, gradient, bias, and/or weighting, used, generated, and/or updated by the image processing model 102 to assist in adjusting the resolution of the target image from the first resolution to the second resolution in the process of processing the target image. Thereafter, in the process of processing the target image, the image processing model 102 may quickly write, read, and/or update the aforementioned various (transient) data by accessing the reserved space in the target memory.
In an embodiment, in the target memory, the total number of reserved spaces disposed by the processor 15 may correspond to or match the type of data used by the image processing model 102 to be stored in the process of processing the target image. For example, in an embodiment, assuming that the data used by the image processing model 102 in the process of processing the target image includes the aforementioned four types of data (that is, tensor, gradient, bias, and weighting), the total number of reserved spaces disposed in the target memory may be four. In this way, in the subsequent process of processing the target image by the image processing model 102, each reserved space in the target memory may be dedicated to storing a specific type of data corresponding to the reserved space. In addition, the total number of reserved spaces disposed in the target memory may also be increased or decreased according to practical requirements, which is not limited by the disclosure.
In an embodiment, the target memory is the buffer memory with the highest or relatively high usage rate in the buffer memories 14(1)˜14(n) during the execution of the application 101. Therefore, during the subsequent processing of the target image by the image processing model 102, it may be ensured as much as possible that the buffer memory accessed by the application 101 and the image processing model 102 is the same buffer memory in the buffer memories 14(1)˜14(n), thereby reducing the occurrence of data copying of cross-buffer memory (the data copying of the cross-buffer memory may reduce the working efficiency of the image processing model 102). In this way, the processing efficiency of the image processing model 102 for the target image may be effectively improved.
In an embodiment, instead of allowing the image processing model 102 to randomly use any buffer memory during the image processing, the target memory is selected by the aforementioned method, and the reserved space is disposed in the target memory to be dedicated to accessing by the image processing model 102, which improves the work efficiency of the image processing model 102 by about 20% or even higher.
It should be noted that in the embodiment of
On the other hand, in the process of processing the target image, a hardware device (such as GPU, VPU, and/or NPU) where the computing core 24 is located may also access the target memory 23 to assist the image processing model 102 in performing the relevant neural network computing (such as parameter updating), thereby generating the target image with the second resolution (that is, the image with a high resolution). After generating the target image with the second resolution, the computing core 24 may return the target image with the second resolution to the application 101.
In an embodiment, instead of arbitrarily using the buffer memories 14(1)˜14(n) to temporarily store data in the process of processing the target image, the buffer memory 14(i) is disposed as the target memory 23, which may effectively improve the processing efficiency of the image processing model 102 (and the computing core 24) on the target image.
In an embodiment, assuming that the hardware limitation of the image capturing interface 11 is limited, the target image with the relatively low resolution (such as 640×360) is only captured in the slow motion photographing (or videoing) mode provided by the application 101 based on a relatively high frame rate (such as 120 FPS). After a very fast procedure of resolution magnification (as shown in
Returning to
In an embodiment, compared to the hardware limitation of the image capturing interface 11 traditionally limited and the image with the low resolution (such as the image with the resolution of 640×360) only presented in the slow motion videoing and playing mode based on the relatively high frame rate (such as 120 FPS), in the embodiments of the disclosure, in the slow motion playing mode, the display interface 12 may still present the image with the high resolution (such as the image with the resolution of 1920×1080 or 2560×1440) based on the aforementioned relatively high frame rate (such as 120 FPS), thereby effectively improving an experience of the user.
However, each step in
In summary, the image resolution adjustment method and the electronic device provided by the embodiments of the disclosure may break through the hardware limitations of the image capturing interface. In the specific photographing mode (such as the slow motion videoing mode), the image resolution adjustment method and the electronic device may quickly improve the resolution of the captured target image with the low resolution based on a high frame rate, and the target image with increased resolution may be played (for example, played in the slow motion) in real time based on the high frame rate, thereby effectively improving the experience of the user.
Although the disclosure has been described in detail with reference to the above embodiments, they are not intended to limit the disclosure. Those skilled in the art should understand that it is possible to make changes and modifications without departing from the spirit and scope of the disclosure. Therefore, the protection scope of the disclosure shall be defined by the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 112151708 | Dec 2023 | TW | national |
