Patent Application
20220207747
This application relates to image processing technologies, and in particular, to an image distortion correction method and an apparatus.
With the development of information technology, increasing functions are integrated into a terminal (for example, a mobile phone), and a photographing function becomes one of important functions of the terminal. In recent years, as there is an increasingly large field of view of a lens of the terminal, there is an increasingly significant problem of portrait distortion in an image captured by the terminal.
Image distortion refers to a phenomenon in which deformation/distortion, in a geometric shape, such as stretching or distortion occurs in content of an image and that is caused by the fact that an optical system with a large field of view and a short focal length does not meet a pinhole model, and after chief rays of different fields of view pass through the optical system, a height of a focal point on a Gaussian image plane is not equal to an ideal image height, and consequently an actual image point deviates from an ideal image point. For example, for a mobile phone with a wide-angle lens, an image of a large field of view can be obtained, but significant image distortion can be easily introduced. For example, in a captured image, more severe deformation/distortion is presented for a portrait (namely, a portrait of a large field of view) located on an outer side of the image. Therefore, an image in which image distortion occurs needs to be corrected, so that the image fits human vision.
In an existing solution, a rectangular object region and a background region are determined by using a detection algorithm in a source image (original image) collected by a camera, and a homography constraint is established for the rectangular object region, to correct stretching deformation occurring in the rectangular object region; a straight line region in the background region is determined by using a straight line detection algorithm, and a straight line constraint is established for the straight line region, to ensure that a corrected straight line is not curved; and a homography compatibility constraint is further established for the rectangular object region and the background region, to ensure continuity of a straight line boundary between the two regions, so as to obtain a final corrected image through rendering.
However, in this solution, a straight line needs to be calibrated on the source image, and a final image correction effect is affected by straight line detection precision. In addition, in this solution, a portrait can be corrected only as a whole, and consequently a final portrait does not really fit human vision. As shown in
Therefore, how to implement natural image distortion correction is still a difficult technical challenge.
Embodiments of this application provide an image distortion correction method and an apparatus, to resolve a disadvantage in the conventional technology, implement natural image distortion correction, and improve user experience.
According to a first aspect, an embodiment of this application provides an image distortion correction method. The method may be applied to a terminal, and the method includes: performing optical distortion correction on a collected source image, to obtain a first corrected image, where the first corrected image includes a background region and a portrait in which stretching deformation occurs, and the portrait in which stretching deformation occurs includes at least a first human body region in which stretching deformation occurs and a second human body region in which stretching deformation occurs; and performing algorithm constraint correction on the first human body region, the second human body region, and the background region, to obtain a second corrected image, where constraint terms used for the algorithm constraint correction respectively constrain the first human body region, the second human body region, and the background region. Specifically, the constraint terms corresponding to the first human body region and the second human body region are determined based on different projection algorithms, so that different deformation amounts can be achieved in the first human body region and the second human body region, to correct the stretching deformation in both the first human body region and the second human body region. The constraint term of the background region is used to constrain the background region in the second corrected image, so that image content in the background region is not distorted, for human vision there is no discontinuity in image content throughout the portrait in the background region, and a smooth and natural transition is implemented between the first human body region, the second human body region, and the background region.
It may be learned that in the solution of this application, the terminal may perform, by region (including the first human body region, the second human body region, and the background region), algorithm constraint correction processing on an image (namely, the first corrected image) obtained after optical distortion correction, and separately perform shape correction based on different projection algorithms when constraining the first human body region and the second human body region of the portrait, to ensure that the portrait is best corrected. A background image is constrained, to ensure that no background distortion or discontinuity occurs in the image content in the background region due to foreground correction. Therefore, implementing the solution of this application can ensure that after the image is corrected, both the portrait and background content are natural, continuous, and harmonious, and fit a human visual habit. In this way, user experience is greatly improved.
According to the first aspect, in a possible implementation, the first corrected image further includes a field of view edge region, and the field of view edge region is a strip region located on an image edge in the first corrected image; and the performing algorithm constraint correction on the first human body region, the second human body region, and the background region, to obtain a second corrected image includes: performing algorithm constraint correction on the first human body region, the second human body region, the background region, and the field of view edge region, to obtain the second corrected image.
During implementation of this embodiment of this application, boundary constraint is performed for the field of view edge region, and therefore a problem of a field of view loss can be avoided, and a content loss caused by cropping can be reduced. In this way, user experience is improved.
According to the first aspect, in a possible implementation, the first human body region includes a head region of the portrait, and the second human body region includes a body region of the portrait.
The head region and the body region of the portrait differ in stretching deformation amount and deformation manner, and therefore the head region and the body region differ in area, and a user has different tolerance for deformation of the head region and the body region. During implementation of this embodiment of this application, algorithm constraint correction processing may be performed on the head region and the body region of the portrait, the background region, and the field of view edge region, so that optimization can be performed for deformation of both the head region and the body region, to ensure that the portrait is best corrected.
According to the first aspect, in a possible implementation, the performing algorithm constraint correction on the first human body region, the second human body region, the background region, and the field of view edge region, to obtain the second corrected image includes: determining initial image matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region in the first corrected image; constructing constraint terms respectively corresponding to the head region, the body region, the background region, and the field of view edge region; constructing a regular constraint term for the first corrected image; and obtaining displacement matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region based on the constraint terms respectively corresponding to the head region, the body region, the background region, and the field of view edge region, the regular constraint term, and weight coefficients corresponding to the constraint terms; and obtaining the second corrected image through color mapping based on the initial image matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region and the displacement matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region.
Specifically, the terminal may set the weight coefficients for the constraint terms of the head region and the body region of the portrait, the field of view edge region, and the background region and the regular constraint term, establish a constraint equation based on the constraint terms and the corresponding weight coefficients, and solve the constraint equation, to obtain an offset of each pixel in each region. A new position of each pixel in a final corrected image (namely, the second corrected image) may be known based on an original position and the offset of each pixel, and the second corrected image may be obtained by performing color mapping at the new position. Therefore, during implementation of this embodiment of this application, the image can be automatically corrected, and an algorithm constraint correction process can be basically performed in a processor of the terminal without being perceived by the user. Therefore, the user can view a rectangular image effect with both an excellent portrait effect and an excellent background effect after completing photographing by using the terminal, thereby greatly improving user experience.
According to the first aspect, in a possible implementation, the obtaining the second corrected image through color mapping based on the corresponding initial image matrices and the corresponding displacement matrices includes: obtaining target image matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region based on the initial image matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region and the displacement matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region; and obtaining the second corrected image based on the target image matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region and color matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region.
According to the first aspect, in a possible implementation, it is assumed that a coordinate matrix (which may also be referred to as a target image matrix) of the image (namely, the second corrected image) obtained after algorithm constraint correction is Mt(i, j), where Mt(i, j)=[ut(i, j), vt(i, j)]T.
A displacement matrix of the second corrected image with respect to an image matrix M0(i, j) in the first corrected image is Dt(i, j), where Dt(i, j)=[du(i, j), dv(i, j)]T.
In other words, Mt(i, j)=M0(i, j)+Dt(i, j).
The weight coefficients are assigned to the constraint terms, and the following constraint equation is constructed: Dt(i, j)=(du(i, j), dv(i, j))=argmin(α1(i, j)*Term1(i, j)+α2(i, j)*Term2(i, j)+α3 (i, j)*Term3(i, j)+α4(i, j)*Term4(i, j)+α5(i, j)*Term5(i, j)).
Herein, Term1 to Term5 are constraint terms respectively corresponding to the head region, the body region, the field of view edge region, the background region, and a global image region, and α1 (i, j) to α5(i, j) are weight coefficients (weight matrices) respectively corresponding to Term1 to Term5.
A displacement matrix Dt(i, j) of each pixel in the image may be obtained by solving the constraint equation by using a least square method, a gradient descent method, or various improved algorithms.
The constraint term (Term1) corresponding to the head region is used to constrain the target image matrix corresponding to the head region to approximate an image matrix obtained after geometric transformation is performed on a first image matrix, to correct stretching deformation occurring in the head region. The first image matrix is obtained after the initial image matrix corresponding to the head region is processed by using a spherical projection algorithm. The geometric transformation includes at least one of image rotation, image translation, and image scaling.
The constraint term (Term2) corresponding to the body region is used to constrain the target image matrix corresponding to the body region to approximate an image matrix obtained after geometric transformation is performed on a second image matrix, to correct stretching deformation occurring in the body region. The second image matrix is obtained after the initial image matrix corresponding to the body region is processed by using a Mercator projection algorithm. The geometric transformation includes at least one of image rotation, image translation, and image scaling.
In this embodiment, the head region is corrected by using the spherical projection algorithm, so that the head of the portrait in the image looks natural and harmonious in human vision. The body region is corrected by using the Mercator projection algorithm, so that the body of the portrait in the image looks straight and harmonious, and is not inclined in human vision. The spherical projection algorithm is combined with the Mercator projection algorithm, so that the entire portrait fits human vision, and the portrait looks natural, harmonious, and non-obtrusive, thereby improving user experience.
It should be noted that in this application, coordinate processing may be performed for the head region and the body region of the portrait by using other projection algorithms. For example, an evolved adaptive spherical projection algorithm may be used for the head region, and an adaptive Mercator algorithm may be used for the body region.
The constraint term (Term3) corresponding to the field of view edge region is used to constrain a pixel in the initial image matrix corresponding to the field of view edge region to be displaced along an edge of the first corrected image or towards an outer side of the first corrected image, to maintain or expand the field of view edge region.
The constraint term (Term4) corresponding to the background region is used to constrain a pixel in the initial image matrix corresponding to the background region to be displaced and a first vector corresponding to the pixel before displacement and a second vector corresponding to the pixel after displacement to be parallel to each other, so that the image content in the background region is smooth and continuous, and the image content throughout the portrait in the background region is continuous and consistent in human vision. The first vector represents a vector between the pixel before displacement and a neighboring pixel corresponding to the pixel before displacement, and the second vector represents a vector between the pixel after displacement and a neighboring pixel corresponding to the pixel after displacement.
The regular constraint term (Term5) is used to constrain a difference between any two of the displacement matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region in the second corrected image to be less than a preset threshold, so that global image content of the second corrected image is smooth and continuous.
It may be learned that in this embodiment of this application, when performing, by region (including the head region, the body region, the background region, and the field of view edge region), algorithm constraint correction processing on the abnormal-shaped image obtained after optical distortion correction, the terminal separately sets independent constraint terms to optimize a solution, to ensure that the portrait is best corrected. For the background region, a position relationship between a background pixel and a pixel in a control region of the background pixel before and after transformation is constrained, to ensure that no background distortion or discontinuity occurs in the image content in the background region due to foreground correction. For the field of view edge region, content adaptive boundary control is used. In this way, there is no problem of a field of view loss, and a field of view boundary can even be expanded, to minimize a content loss caused by cropping and the problem of a field of view loss. Therefore, implementing this embodiment of this application can ensure that after the image is corrected, both the portrait and the background content are natural, continuous, and harmonious, and fit a human visual habit. In this way, user experience is greatly improved, and there are great commercial application benefits.
According to the first aspect, in a possible implementation, the first corrected image is a non-rectangular image; and after the obtaining the second corrected image, the method further includes: cropping the second corrected image, to obtain a rectangular image, and outputting the rectangular image.
According to the first aspect, in a possible implementation, the performing optical distortion correction on a collected source image, to obtain a first corrected image includes: performing optical distortion correction on the collected source image, to obtain a non-rectangular image; and cropping the non-rectangular image into a rectangular image, where the first corrected image is the rectangular image.
According to the first aspect, in a possible implementation, an image area occupied by the first human body region is positively correlated with a weight coefficient corresponding to a constraint term of the first human body region; and an image area occupied by the second human body region is positively correlated with a weight coefficient corresponding to a constraint term of the second human body region.
Specifically, an image area occupied by the head region of the portrait in the first corrected image may be configured to be positively correlated with the weight coefficient corresponding to the head constraint term, and an image area occupied by the body region of the portrait in the first corrected image may be configured to be positively correlated with the weight coefficient corresponding to the body constraint term. That is, when the head region or the body region occupies a large image area, a weight value of the head constraint term (or called as face constraint term) or the body constraint term of the corresponding region may be increased because a portrait with a large area is more sensitive to deformation, so that the head region or the body region is more natural and harmonious after correction.
According to the first aspect, in a possible implementation, a field of view of the first human body region is positively correlated with the weight coefficient corresponding to the constraint term of the first human body region; and a field of view of the second human body region is positively correlated with the weight coefficient corresponding to the constraint term of the second human body region.
Specifically, a field of view of the head region in the first corrected image may be configured to be positively correlated with the weight coefficient corresponding to the head constraint term, and a field of view of the body region in the first corrected image may be configured to be positively correlated with the weight coefficient corresponding to the body constraint term. That is, when the head region or the body region is a region with a large field of view, the weight value of the head constraint term or the body constraint term of the corresponding region may be increased because a portrait with a large field of view tends to be more severely deformed, so that the head region or the body region is more natural and harmonious after correction, to avoid significant deformation.
According to the first aspect, in a possible implementation, texture significance of the background region is positively correlated with the weight coefficient corresponding to the constraint term of the background region.
Specifically, the texture significance of the background region in the first corrected image may be configured to be positively correlated with the weight coefficient corresponding to the background constraint term. That is, when there are weak texture details for the background region, a weight of the background constraint term of the corresponding region may be decreased because the background region is insensitive to distortion or a fault caused by correction, to better correct the foreground portrait.
According to the first aspect, in a possible implementation, a weight coefficient corresponding to the constraint term of the field of view edge region when there is portrait content in the field of view edge region is greater than a weight coefficient corresponding to the constraint term of the field of view edge region when there is no portrait content in the field of view edge region.
Specifically, the weight coefficient corresponding to the field of view edge constraint term when there is portrait content in the field of view edge region in the first corrected image may be configured to be greater than the weight coefficient corresponding to the field of view edge constraint term when there is no portrait content in the field of view edge region. That is, when there is a portrait on a field of view edge of the image, a weight value of the edge constraint term of the corresponding region may be increased because boundary constraint may affect a correction effect of the foreground portrait, to better correct the foreground portrait.
According to the first aspect, in a possible implementation, that the terminal performs optical distortion correction on a source image, to obtain a first corrected image includes: performing, by using an optical distortion correction algorithm, optical distortion correction on the source image collected by a camera apparatus. Specifically, the optical distortion correction algorithm may be a Zhang Zhengyou's camera calibration algorithm. After processing is performed by using the optical distortion correction algorithm, the first corrected image is obtained, and a curved line is corrected into a straight line in the first corrected image.
According to the first aspect, in a possible implementation, the terminal may divide the first corrected image into object regions such as the first human body region, the second human body region, the background region, and the field of view edge region by using an image segmentation method. For example, the image segmentation method may be an image semantic segmentation method in which pixel-level image recognition is automatically performed on the first corrected image by using a neural network processing unit (NPU) in the terminal, an object category to which each pixel in the first corrected image belongs is marked, and content in the first corrected image is recognized, to divide the first corrected image into object regions.
According to the first aspect, in a possible implementation, the first human body region of the portrait may be a head region and a limb region of the portrait, and the second human body region of the portrait may be a torso region of the portrait.
According to a second aspect, an embodiment of this application provides an apparatus. The apparatus includes an optical distortion correction module and an algorithm constraint correction module. The optical distortion correction module is configured to perform optical distortion correction on a collected source image, to obtain a first corrected image. The first corrected image includes a background region and a portrait in which stretching deformation occurs. The portrait in which stretching deformation occurs includes at least a first human body region in which stretching deformation occurs and a second human body region in which stretching deformation occurs. The algorithm constraint correction module is configured to perform algorithm constraint correction on the first human body region, the second human body region, and the background region, to obtain a second corrected image. Constraint terms used for the algorithm constraint correction respectively constrain the first human body region, the second human body region, and the background region.
The apparatus may specifically implement, by using the functional modules, the method described in any implementation/embodiment of the first aspect.
According to a third aspect, an embodiment of this application provides a terminal. The terminal may include one or more processors and a memory. The processor and the memory are connected or coupled by using a bus. The memory stores one or more computer programs. The one or more processors are configured to execute the one or more computer programs. The one or more computer programs include instructions, and the instructions are used to perform the method described in the possible implementation/embodiment of the first aspect.
According to a fourth aspect, an embodiment of this application provides a terminal. The terminal may include one or more processors, a memory, and a display screen configured to display an image.
The processor, the memory, and the display screen may be connected by using one or more communications buses. The memory stores one or more computer programs. The one or more processors are configured to execute the one or more computer programs. The one or more computer programs include instructions, and the instructions are used to perform the method described in the possible implementation/embodiment of the first aspect.
According to a fifth aspect, an embodiment of this application provides a computer-readable storage medium that stores a computer program, where the computer program includes program instructions. When the program instructions are executed by a processor, the processor is enabled to perform the method described in any one of the embodiments of the first aspect.
According to a sixth aspect, an embodiment of this application provides a computer program product. When the computer program product runs on a computer, the method described in any one of the embodiments of the first aspect is implemented.
It may be learned that in the embodiments of this application, the terminal may perform, by region (including the head region, the body region, the background region, and the field of view edge region), algorithm constraint correction processing on the image obtained after optical distortion correction, and separately perform shape correction on the head region and the body region of the portrait based on different projection algorithms, to ensure that the portrait is best corrected. In the embodiments of this application, it can be further ensured that no background distortion or discontinuity occurs in the image content in the background region due to foreground correction, and a content loss caused by cropping and the problem of a field of view loss can be reduced. Therefore, implementing the embodiments of this application can ensure that after the image is corrected, both the portrait and the background content are natural, continuous, and harmonious, and fit a human visual habit. In this way, user experience is greatly improved.
Various implementations of this application are described in detail below based on the accompanying drawings. It should be understood that this specification is not intended to limit this application to those example implementations. Rather, this application is intended to cover these implementations, and various alternatives, modifications, equivalents, and other implementations that may be included within the spirit and scope of this application as defined by the appended claims.
In the descriptions of the embodiments of this application, unless otherwise specified, “I” indicates “or”. For example, AB may indicate A or B. The term “and/or” in this specification merely describes an association relationship for describing associated objects, and indicates that three relationships may exist. For example, A and/or B may indicate the following three cases: Only A exists, both A and B exist, and only B exists. In addition, in the descriptions of the embodiments of this application, “a plurality of” means two or more.
The following terms “first” and “second” are merely intended for description, and shall not be understood as an indication or implication of relative importance or implicit indication of a quantity of indicated technical features. Therefore, a feature limited by “first” or “second” may explicitly or implicitly include one or more features. In the descriptions of embodiments of this application, unless otherwise specified, “a plurality of” means two or more.
A method in the embodiments of this application may be applied to a terminal device (terminal for short). The terminal may be a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a mobile internet device (MID, mobile internet device), a wearable device (for example, a smart band or a smart watch), an AR device, a VR device, a camera device (for example, a video recorder, a smart camera, a digital camera, or a camera), a vehicle-mounted device, or another device with a photographing function.
The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, a neural-network processing unit (neural-network processing unit, NPU), and/or the like. Different processing units may be independent components, or may be integrated into one or more processors.
A memory may be further disposed in the processor 110, and is configured to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data just used or cyclically used by the processor 110. If the processor 110 needs to use the instructions or the data again, the processor 110 may directly invoke the instructions or the data from the memory, to avoid repeated access and reduce waiting time of the processor 110, thereby improving system efficiency.
In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (inter-integrated circuit, I2C) interface, an inter-integrated circuit sound (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver/transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (general-purpose input/output, GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, a universal serial bus (universal serial bus, USB) interface, and/or the like.
It can be understood that an interface connection relationship, between the modules, shown in this embodiment of this application is merely an example, and does not constitute a limitation on a structure of the terminal 100. In some other embodiments of this application, the terminal 100 may alternatively use an interface connection manner different from that in this embodiment, or a combination of a plurality of interface connection manners.
The processor 110 may read an instruction, control execution of the instruction, and invoke data based on instruction operation code and a sequence signal. Specifically, the processor 110 may be configured to perform an image distortion correction method described in the embodiments of this application.
The charging management module 140 is configured to receive a charging input from the charger. The charger may be a wireless charger or a wired charger.
The power management module 141 is configured to connect to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives an input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, an external memory, the display 194, the camera apparatus 193, the wireless communications module 160, and the like.
A wireless communication function of the terminal 100 may be implemented by using the antenna 1, the antenna 2, the mobile communications module 150, the wireless communications module 160, the modem processor, the baseband processor, and the like.
The terminal 100 may implement, by using the GPU, the display 194, the application processor, and the like, a function of displaying an image/video. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is configured to perform mathematical and geometric calculation, and render an image. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
The display 194 is configured to display an image, a video, and the like. The display 194 includes a display panel. The display panel may be a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (organic light-emitting diode, OLED), an active-matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), a flex light-emitting diode (flex light-emitting diode, FLED), a mini LED, a micro LED, a micro-OLED, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), or the like. In some embodiments, the terminal 100 may include one or N displays 194, where N is a positive integer greater than 1.
In some implementations, the display 194 may be a touchscreen. The touchscreen may include a display panel and a touch-sensitive surface that covers the display panel. When detecting a touch operation (for example, an operation such as touching, tapping, pressing, or sliding performed by a user on or near the touch-sensitive surface by using any suitable object or accessory such as a finger or a stylus) on or near the touch-sensitive surface, the touch-sensitive surface transfers the touch operation to the processor 110 to determine a type of a touch event, and then the processor 110 provides corresponding visual output on the display panel based on the type of the touch event. In an example, the touch-sensitive surface and the display panel are used as two independent components to implement input and input functions. In still another example, the touch-sensitive surface and the display panel are integrated to implement input and output functions.
Optionally, the touch-sensitive surface may include two parts: a touch detection apparatus and a touch controller. The touch detection apparatus detects a touch position of the user, detects a signal brought by a touch operation, and transfers the signal to the touch controller. The touch controller receives touch information from the touch detection apparatus, converts the touch information into touch point coordinates, and sends the touch point coordinates to the processor 110, and can receive and execute a command sent by the processor 110. In addition, the touch-sensitive surface may be implemented in a plurality of types such as a resistive type, a capacitive type, an infrared type, and a surface acoustic wave type.
Optionally, the display panel may be configured to display information entered by the user or information provided for the user and various display interfaces of the terminal 100. The display interface may be a user interface (User Interface, UI) or a graphical user interface (Graphical User Interface, GUI). Interface content may include an interface of an application that is running, a system-level menu, and the like, and may specifically include an image (picture), text (text), an icon (Icon), a video (video), a button (Button), a slider bar (Scroll Bar), a menu (Menu), a window (Window), a label (Label), an input box (input box), and any combination thereof.
The terminal 100 may implement a photographing function through the ISP, the camera apparatus 193, the video codec, the GPU, the display 194, the application processor, and the like.
The ISP is configured to process data fed back by the camera apparatus 193. For example, during photographing, a shutter is pressed, light is transmitted to a photosensitive element of the camera through a lens, an optical signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, to convert the electrical signal into a visible image. The ISP may further perform algorithm optimization on noise, brightness, and complexion of the image. The ISP may further optimize parameters such as exposure and a color temperature of a photographing scene. In some embodiments, the ISP may be disposed in the camera apparatus 193.
The camera apparatus 193 is configured to capture a static image or a video. An optical image is generated after an object passes through the lens (for example, an optical lens module), and is projected to the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a complementary metal-oxide-semiconductor (complementary metal-oxide-semiconductor, CMOS) phototransistor. The photosensitive element converts an optical signal into an electrical signal, and then transmits the electrical signal to the ISP for converting the electrical signal into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. Then, the image signal may be displayed on the display 194, and/or the image signal may be stored in the internal memory 121 or the external memory.
In some embodiments, the camera apparatus 193 may include a camera, for example, an infrared camera or another camera, for collecting an image required for facial recognition. The camera for collecting an image required for facial recognition is usually located on a front side of an electronic device, for example, above a touchscreen, or may be located at another position. This is not limited in this embodiment of this application.
In some embodiments, the camera of the camera apparatus 193 includes but is not limited to an optical camera, an infrared camera, and a depth camera. A specific form may be a single-lens camera or a multi-lens camera. A lens of the camera may be a standard lens, a wide-angle lens, an ultra-wide-angle lens, a fisheye lens, a long-focus lens, or a combination thereof.
In some embodiments, the camera of the camera apparatus 193 may include a front-facing camera and/or a rear-facing camera.
In this specification, an image signal output by the camera apparatus 193 may be referred to as a “source image”. The source image may be output to the processor for further image distortion correction processing.
The video codec is configured to compress or decompress a digital video. The terminal 100 may support one or more video codecs. Therefore, the terminal 100 may play or record videos in a plurality of coding formats, for example, moving picture experts group (moving picture experts group, MPEG)1, MPEG2, MPEG3, and MPEG4.
The DSP is configured to process a digital signal. In addition to the digital image signal, the DSP may further process another digital signal. For example, when the terminal 100 selects a frequency, the digital signal processor is configured to perform Fourier transform and the like on frequency energy.
The NPU is configured to quickly process input information by using a biological neural network structure such as a mode of transmission between human neurons, and may further constantly perform self-learning. The NPU can implement applications such as intelligent cognition of the terminal 100, such as image recognition, facial recognition, speech recognition, and text understanding.
The internal memory 121 may be configured to store computer-executable program code. The executable program code includes instructions. The processor 110 runs the instructions stored in the internal memory 121, to implement various function applications and data processing of the terminal 100. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, an application (for example, an app for camera shooting or an app for image beautifying) required by at least one function, and the like. The data storage area may store data (for example, corrected image data) and the like created when the terminal 100 is used. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory, for example, at least one magnetic disk storage device, a flash memory, and a universal flash storage (universal flash storage, UFS).
The external memory interface 120 may be configured to connect to an external memory card, for example, a micro SD card, to expand a storage capability of the terminal 100. The external storage card communicates with the processor 110 through the external memory interface 120, to implement a data storage function. For example, files such as music and a video are stored in the external memory card.
The terminal 100 may implement audio functions such as music playing, video background music playing, and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headset jack 170D, the application processor. The audio module 170 is configured to convert digital audio information into analog audio signal output, and is also configured to convert analog audio input into a digital audio signal. The speaker 170A, also referred to as a “loudspeaker”, is configured to convert an audio electrical signal into a sound signal. The receiver 170B, also referred to as an “earpiece”, is configured to convert an audio electrical signal into a sound signal. The microphone 170C, also referred to as a “mike” or a “mic”, is configured to convert a sound signal into an electrical signal. The headset jack 170D is configured to connect to a wired headset. The headset jack 170D may be a USB port 130, or may be a 3.5 mm open mobile terminal platform (open mobile terminal platform, OMTP) standard interface or cellular telecommunications industry association of the USA (cellular telecommunications industry association of the USA, CTIA) standard interface.
The button 190 includes a power button, a volume button, and the like. The button 190 may be a mechanical button, or may be a touch button. The terminal 100 may receive a key input, and generate a key signal input related to user setting and function control of the terminal 100.
The indicator 192 may be an indicator light, and may be configured to indicate a charging status and a power change, or may be configured to indicate a message, a missed call, a notification, and the like.
The SIM card interface 195 is configured to connect to a SIM card. The SIM card may be inserted into the SIM card interface 195 or detached from the SIM card interface 195, to implement contact with or separation from the terminal 100. In some embodiments, the terminal 100 uses an eSIM, namely, an embedded SIM card. The eSIM card may be embedded into the terminal 100, and cannot be separated from the terminal 100.
It can be understood that the structure shown in this embodiment of this application does not constitute a specific limitation on the terminal 100. In some other embodiments of this application, the terminal 100 may include more or fewer components than those shown in the figure, or combine some components, or split some components, or have different component arrangements. Components shown in figure may be implemented in hardware, software, or a combination of hardware and software that includes one or more signal processing and/or application-specific integrated circuits.
A software system of the terminal 100 may use a layered architecture, an event-driven architecture, a microkernel architecture, a micro service architecture, or a cloud architecture. A software structure of the terminal 100 is described below by using an Android (Android) system with a layered architecture as an example.
In a layered architecture, software is divided into several layers, and each layer has a clear role and task. The layers communicate with each other through a software interface. In some embodiments, an Android system is divided into four layers: an application layer, an application framework layer, an Android runtime (Android runtime) and system library, and a kernel layer from top to bottom.
The application layer may include a series of application packages. As shown in
The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for an application at the application layer. The application framework layer includes some predefined functions. As shown in
The window manager is configured to manage a window program. The window manager may obtain a size of a display, determine whether there is a status bar, perform screen locking, and the like.
The content provider is configured to: store and obtain data, and enable the data to be accessed by an application. The data may include image data, video data, and the like.
The resource manager provides various resources for an application, such as a localized portrait string, an icon, a picture, a layout file, and a video file.
The view system includes visual controls, such as a control for displaying a text and a control for displaying an image. The view system may be configured to construct a display interface of an application.
For example, a display interface of a photographing app presented by using the view system is shown in
The preview box 204 is used to preview a scene image to be captured.
In an embodiment of this application, an image presented in the preview box 204 may be a corrected image (which may also be referred to as a first corrected image) obtained after optical distortion correction processing is performed.
In still another embodiment of this application, an image presented in the preview box 204 may be a corrected image (which may also be referred to as a second corrected image) obtained after optical distortion correction processing and algorithm constraint processing are performed.
In still another embodiment of this application, an image presented in the preview box 204 may be a source image that does not undergo optical distortion correction processing or algorithm constraint processing.
When tapping or touching the switching control 203 for performing switching between the front-facing camera and the rear-facing camera, the user may indicate the terminal 100 to select the front-facing camera or the rear-facing camera for photographing.
When the user taps or touches the photographing control 202, the terminal 100 drives a camera apparatus to initiate a photographing operation, indicates the lower-layer system library to process a source image into a second corrected image, and stores the second corrected image in an album.
When the user taps or touches the image browsing control 201, the terminal 100 may invoke the album app, and display the second corrected image. As shown in
Scheduling and management of the Android system during Android runtime (Android Runtime) may include a kernel library and a virtual machine. The kernel library includes two parts: a function that needs to be invoked in Java language and a kernel library of Android.
The application layer and the application framework layer run on the virtual machine. The virtual machine executes Java files at the application layer and the application framework layer as binary files. The virtual machine is configured to implement functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.
The system library may include a plurality of functional modules, for example, a surface manager, a media library, and a graphics engine (for example, SGL).
The surface manager is configured to: manage a display subsystem, and provide a layer fusion function for a plurality of applications.
The media library supports playback and recording of audio and video in a plurality of commonly used formats, static image files, and the like. The media library may support a plurality of audio and video coding formats, such as MPEG-4, H.264, MP3, AAC, AMR, JPG, and PNG.
The graphics engine is a drawing engine for performing image processing. In this embodiment of this application, the graphics engine may be configured to: process a source image into a first corrected image, and process the first corrected image into a second corrected image.
The kernel layer is a layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver. The camera driver may be configured to drive the camera of the terminal 100 to perform photographing. The display driver may be configured to display an image (the first corrected image or the second corrected image) on a display panel of a display screen.
The following describes some concepts in the embodiments of this application.
In the embodiments of this application, a “source image” represents an original image collected by a camera apparatus of a terminal. The camera apparatus includes an optical lens module and a photosensitive element. For example, the photosensitive element includes a charge coupled device (CCD). During photographing, light of a to-be-photographed object is focused by using the optical lens module and projected to the CCD. The CCD is made of a highly photosensitive semiconductor material, and usually includes many photosensitive units that are usually in megapixels. When a surface of the CCD is illuminated by light, each photosensitive unit reflects a charge on a component, to convert the light into an electrical signal. Signals generated by all the photosensitive units together form a complete electrical signal. The CCD then transfers the electrical signal to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV.
An imaging process of the camera apparatus essentially reflects a conversion process of a coordinate system. That is, the imaging process is actually a process of converting a point of the to-be-photographed object into a world coordinate system to a camera coordinate system of the camera apparatus, then projecting the point to a physical image coordinate system of an imaging plane, and finally converting data on the imaging plane into an image pixel coordinate system of an image plane.
Due to an inherent characteristic (light is converged by a convex lens and light is diffused by a concave lens) of a lens in the optical lens module, lens precision, and a processing technology, optical distortion is usually introduced into a source image obtained by using the optical lens module, resulting in image distortion. Distortion refers to a degree of distortion of an image, obtained by using an optical system, of an object with respect to the object. Optical distortion refers to a degree of deformation calculated based on an optical theory. A spatial straight line is curved in an image in which optical distortion occurs. Distortion caused by optical distortion is very unfavorable to imaging quality of a photo. Due to existence of optical distortion, a curved line may be presented after a straight line of the to-be-photographed object in the world coordinate system is finally converted into the image pixel coordinate system. Particularly, in an image, optical distortion causes more significant distortion to the image at a position closer to an edge of the image. In addition, there is more significant optical distortion for a wide-angle camera than for an ordinary camera.
When the terminal is a mobile terminal such as a mobile phone, a tablet computer, or a wearable device, the optical lens module usually suffers from severe optical distortion due to an optical design of the lens. This is related to a field of view of the mobile terminal. A larger field of view usually indicates more severe optical distortion suffered by the optical lens module.
Optical distortion may also be referred to as image distortion, lenticular lens distortion, shooting distortion, camera distortion, lens distortion, and the like. Types of optical distortion mainly include radial distortion and tangential distortion.
Tangential distortion usually occurs due to the fact that the lens is not parallel to the image plane because there is a manufacturing disadvantage for the camera apparatus.
Radial distortion occurs due to the fact that light is curved to a larger extent at a position far away from a center of the lens than at a position close to the center. Specifically, radial distortion mainly includes two types: barrel distortion and pincushion distortion.
Barrel aberration may also be referred to as barrel distortion, and is a distortion phenomenon that is caused due to physical properties of the lens and a structure of a lens group and in which an imaging picture presents barrel-shaped expansion. Usually, it is most likely to detect the barrel distortion phenomenon when a wide-angle lens or a wide-angle end with a zoom lens is used. It is most likely to detect barrel distortion when there is a line (especially a line close to an edge of an image) in a picture.
Pincushion aberration may also be referred to as pincushion distortion, and is a distortion phenomenon that is caused due to the lens and in which a picture “converges” towards a center. Usually, it is most likely to detect the pincushion distortion phenomenon when a long-focus lens or a long-focus end with a zoom lens is used. Pincushion distortion is prone to occur especially when a focal length converter is used. It is most likely to detect pincushion distortion when there is a line (especially a line close to an edge of an image) in a picture.
Therefore, in the embodiments of this application, the “source image” collected by the camera apparatus may also be referred to as a “distorted image”.
To reduce distortion in an image, in the embodiments of this application, optical distortion correction may be first performed on the source image. During optical distortion correction, a mapping relationship between point positions before and after image distortion is calculated by using an optical distortion correction algorithm, to determine a distortion coefficient of the image, and then transformation opposite to distortion is performed, and a pixel value of a position after distortion is assigned to an original position, to alleviate distortion caused by aberration.
As shown in
In the abnormal-shaped image, background content that is curved due to distortion is usually well corrected, and becomes normal straight line content. However, for a portrait in the image, there is little inherent straight line content, and therefore there is still a deformation/distortion condition such as stretching or distortion, including head deformation, body deformation, and the like in the portrait.
To better correct the deformed portrait, ensure that there is no significant deformation (for example, a background straight line cannot be curved or include a fault) in another region while the portrait is corrected, and ensure that for a corrected image, no excessively large field of view loss is caused due to cropping into a rectangular shape, in the embodiments of this application, algorithm constraint correction continues to be performed on the image (which may be referred to as a first corrected image) obtained after optical distortion correction, and a head and a body part of the portrait, an image background, an image edge, and the like are corrected based on different algorithms, to obtain a second corrected image. Both the head and the body of the portrait in the second corrected image can fit human vision, and image content throughout the portrait in the image background in the second corrected image is continuous and consistent in human vision, to avoid curved content or a fault condition in the image background. In addition, a field of view loss is well avoided for the second corrected image. Content of algorithm constraint correction is described in detail below.
Based on the foregoing description, the following provides some image distortion correction methods provided in the embodiments of this application.
For convenience, the method embodiments described below are expressed as a combination of a series of action steps. However, a person skilled in the art should understand that specific implementations of the technical solutions of this application are not limited to a sequence of the described series of action steps.
S401. The terminal collects a source image.
Specifically, the terminal may collect the source image by using a camera apparatus (including an optical lens module and a photosensitive element). The source image may be a rectangular image. The optical lens module usually suffers from optical distortion (for example, barrel distortion) to some extent, and therefore image content of the collected source image is deformed or distorted.
S402. The terminal performs optical distortion correction on the source image, to obtain a first corrected image.
Specifically, the terminal may perform, by using an optical distortion correction algorithm, optical distortion correction on the source image collected by the camera apparatus. For example, the optical distortion correction algorithm may be a Zhang Zhengyou's camera calibration algorithm. After processing is performed by using the optical distortion correction algorithm, an abnormal-shaped image (namely, a non-rectangular image) is obtained. In an embodiment, the abnormal-shaped image is the first corrected image. In still another embodiment, the abnormal-shaped image may be cropped into a rectangular image, and the rectangular image may be processed into an image of target resolution through scaling. The image of the target resolution is the first corrected image.
S403. The terminal separately processes a first human body region, a second human body region, and a background region, to obtain a second corrected image.
In this embodiment of this application, to better correct a portrait and ensure that no significant distortion occurs in a background in a process of correcting the foreground portrait, the image needs to be first divided into regions.
Specifically, the terminal may divide the first corrected image into several specific object regions of interest by using an image segmentation method. For example, the image segmentation method may be an image semantic segmentation method in which pixel-level image recognition is automatically performed on the first corrected image by using a neural network processing unit (NPU) in the terminal, an object category to which each pixel in the first corrected image belongs is marked, and content in the first corrected image is recognized, to divide the first corrected image into object regions.
In an implementation, regions obtained by dividing the first corrected image include the first human body region and the second human body region of the portrait and the background region. Each of the first human body region of the portrait and the second human body region of the portrait refers to a part of the portrait. For example, as shown in
Correspondingly, the terminal may separately perform algorithm constraint correction processing on the first human body region, the second human body region, and the background region, to obtain the second corrected image. In algorithm constraint correction, the first human body region and the second human body region of the portrait in the second corrected image may be based on different algorithms (for example, the head region of the portrait is processed by using a spherical projection algorithm, and the body region is processed by using a Mercator algorithm) to present different deformation amounts, so that stretching deformation in both the first human body region and the second human body region is corrected to fit human vision, and image content throughout the portrait in the background region in the first corrected image is continuous and consistent in human vision and includes no discontinuity.
In still another implementation, in addition to the first human body region and the second human body region of the portrait, and the background region, regions obtained by dividing the first corrected image further includes a field of view edge region. The field of view edge region is a strip region that is located at an edge of the image and that is in the first corrected image. In specific implementation, a threshold may be set to determine a width of the strip region. Correspondingly, the background region is a valid region other than the first human body region and the second human body region of the portrait, and the field of view edge region in the first corrected image.
Correspondingly, the terminal may separately perform algorithm constraint correction processing on the first human body region, the second human body region, the background region, and the field of view edge region, to obtain the second corrected image. During algorithm constraint correction, the first human body region and the second human body region of the portrait in the second corrected image may be based on different algorithms (for example, the head region of the portrait is processed by using a spherical projection algorithm, and the body region is processed by using a Mercator algorithm) to present different deformation amounts, so that stretching deformation in both the first human body region and the second human body region is corrected to fit human vision, and image content throughout the portrait in the background region in the first corrected image is continuous and consistent in human vision and includes no discontinuity. In addition, boundary constraint is performed for the field of view edge region, and therefore there is no problem of a field of view loss, and image content on a field of view boundary can even be expanded.
It may be learned that in this embodiment of this application, the terminal may perform, by region (including the head region, the body region, the background region, and the field of view edge region), algorithm constraint correction processing on the abnormal-shaped image obtained after optical distortion correction, and separately perform shape correction on the head region and the body region of the portrait based on different projection algorithms, to ensure that the portrait is best corrected. In this embodiment of this application, it can be further ensured that no background distortion or discontinuity occurs in image content in the background region due to foreground correction, and a content loss caused by cropping and the problem of a field of view loss can be reduced. Therefore, implementing this embodiment of this application can ensure that after the image is corrected, both the portrait and background content are natural, continuous, and harmonious, and fit a human visual habit. In this way, user experience is greatly improved.
S501. The terminal collects a source image. For details, refer to the description of S401 in the embodiment in
S502. The terminal performs optical distortion correction on the source image, to obtain a first corrected image.
Specifically, the terminal may perform, by using an optical distortion correction algorithm, optical distortion correction on the source image collected by a camera apparatus. For example, the optical distortion correction algorithm may be a Zhang Zhengyou's camera calibration algorithm. After processing is performed by using the optical distortion correction algorithm, the first corrected image is obtained. The first corrected image herein is no longer a rectangular image, but an abnormal-shaped image (or referred to as a non-rectangular image). A curved line is corrected into a straight line in the image. However, in this case, a portrait in the image is still deformed to some extent. Usually, a larger field of view of the portrait indicates more severe deformation.
To better correct the deformed portrait, ensure that there is no deformation such as curved content or a discontinuity in the background region while the portrait is corrected, and ensure that for a corrected image, no excessively large field of view loss is caused due to cropping into a rectangular shape, in this embodiment of this application, algorithm constraint correction may be further performed on the first corrected image. An obtained image may be referred to as a second corrected image. In this embodiment, a processing process of algorithm constraint correction may include the following steps S503 to S507.
S503. The terminal recognizes that the first corrected image includes a portrait region in which stretching deformation occurs, a background region, and a field of view edge region. The portrait region in which stretching deformation occurs includes at least a head region in which stretching deformation occurs and a body region in which stretching deformation occurs.
In this embodiment of this application, to better correct the portrait and ensure that there is no significant distortion in a background in a process of correcting the foreground portrait, the image needs to be divided into regions.
Specifically, the terminal may divide the first corrected image into several specific object regions of interest by using an image segmentation method. For example, the image segmentation method may be an image semantic segmentation method in which pixel-level image recognition is automatically performed on the first corrected image by using a neural network processing unit (NPU) in the terminal, an object category to which each pixel in the first corrected image belongs is marked, and content in the first corrected image is recognized, to divide the first corrected image into object regions.
As shown in
The head region of the portrait covers the head, neck, and an associated ornament (for example, a hat, headwear, or glasses) of each portrait in the image.
The body region of the portrait covers the torso, limbs, clothes, a pant, shoes, and the like of each portrait in the image.
Three foreground portraits are shown in the abnormal-shaped image presented in
The field of view edge region is an annular strip region that is close to the invalid region, that is located at an edge of the abnormal-shaped image, and that is in the abnormal-shaped image. In specific implementation, a threshold may be set to determine a width of the strip region.
The background region is a valid region other than the head region and the body region of the portrait, and the field of view edge region in the abnormal-shaped image.
It should be noted that in this embodiment of this application, there is no limitation that the portrait is divided into the head region and the body region, and there may be another division manner. For example, the portrait may be alternatively divided into the head region, a limb region, and a torso region.
S504. The terminal determines initial image matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region.
Specifically, the terminal determines an initial position point and color information of a pixel in each of the head region and the body region of each portrait, the background region, and the field of view edge region in the first corrected image. The initial position point of the pixel may be represented by using an initial image matrix (or referred to as an initial coordinate matrix), and the color information of the pixel may be represented by using a color matrix.
The initial image matrix is used to record a position of each pixel in the first corrected image, and plane position coordinates of a pixel [i, j]T in an ith column and a jth row in the first corrected image may be expressed as follows:
M(i,j)=[u(i,j),v(i,j)]T
Herein, u(i, j) represents a horizontal coordinate, and v(i, j) represents a vertical coordinate.
The color matrix is used to record color information of each pixel. A color matrix of the pixel in the ith column and the jth row in the first corrected image may be expressed as follows:
C(i,j)=[r(i,j),g(i,j),b(i,j)]T.
Herein, r(i, j), g(i, j), and b(i, j) respectively represent red color information, green color information, and blue color information.
To facilitate subsequent operation processing, an initial image matrix of each pixel in the first corrected image may be normalized. It is assumed that a width and a height of the first corrected image are respectively W and H. In this case, a normalized coordinate matrix may be expressed as follows:
u0=(j−W/2)/W, v0=(H/2−i)/H.
Based on the normalized coordinate matrix, the normalized initial image matrix (which may also be referred to as an image matrix) of each pixel in the first corrected image may be expressed by using M0. Details are as follows:
M0(i,j)=[u0(i,j), v0(i,j)]T.
The head region and the body region of the portrait differ in stretching deformation amount and deformation manner, and therefore the head region and the body region differ in area, and a user has different tolerance for deformation of the head region and the body region. In this embodiment of this application, to ensure that deformation of both the head region and the body region is well corrected, coordinate processing is further separately performed for the head region and the body region by using different projection algorithms.
In an embodiment, as shown in
Initial image coordinates M0(i, j), where (i, j)∈HeadRegion, of the head region in the first corrected image are mapped based on the spherical projection algorithm, and are converted into a first image matrix M1, and M1 is denoted as M1(i, j)=[u1(i, j), v1(i, j)]T.
Specifically, the spherical projection algorithm for mapping M0(i, j)=[u0, v0]T to M1(i, j)=[u1, v1]T may be expressed as follows:
u1=λ*tan(0.5 atan(r0))*cos(θ0), v1=λ*tan(0.5a tan(r0))*sin(θ0).
Herein, r0=sqrt(u0*u0+v0*v0), θ0=atan(v0/u0), and λ is a scaling coefficient.
Initial image coordinates M0(i, j), where (i, j)∈BodyRegion, of the body region in the first corrected image are mapped based on the Mercator projection algorithm, and are converted into a second image matrix M2, and M2 is denoted as M2(i, j)=[u2(i, j), v2(i, j)]T, and M2(i, j)=(u2, v2).
Specifically, the Mercator projection algorithm for mapping M0(i, j)=[u0, v0]T to M2(i, j)=[u2(i, j), v2(i, j)]T may be expressed as follows:
u2=λ*atan(u0), v2=λ*log(tan(π/4+ϕ)/2)).
Herein, ϕ=atan(v0/sqrt(u0*u0+1)), and λ is a scaling coefficient.
In this embodiment, M1 obtained by mapping M0 of the head region by using the spherical projection algorithm may be used in a subsequent step S505 to correct stretching deformation in the head of the portrait, and M2 obtained by mapping M0 of the body region by using the Mercator projection algorithm may be used in the subsequent step S505 to correct stretching deformation in the body of the portrait.
In this embodiment, the head region is corrected by using the spherical projection algorithm, so that the head of the portrait in the image looks natural and harmonious in human vision. The body region is corrected by using the Mercator projection algorithm, so that the body of the portrait in the image looks straight and harmonious, and is not inclined in human vision. The spherical projection algorithm is combined with the Mercator projection algorithm, so that the entire portrait fits human vision, and the portrait looks natural, harmonious, and non-obtrusive, thereby improving user experience.
It should be noted that in this application, coordinate processing may be performed for the head region and the body region of the portrait by using other projection algorithms. For example, an evolved adaptive spherical projection algorithm may be used for the head region, and an adaptive Mercator algorithm may be used for the body region.
S505. The terminal constructs constraint terms respectively corresponding to the head region, the body region, the background region, and the field of view edge region and a regular constraint term.
In this embodiment of this application, the terminal separately constructs the corresponding constraint terms for the head region and the body region of the portrait, the background region, and the field of view edge region, and constructs the regular constraint term used for the global image. The corresponding constraint terms constructed for the head region and the body region of the portrait, the background region, and the field of view edge region and the regular constraint term are jointly used to determine displacement required by each pixel in the image.
The following separately describes the constraint terms of the head region and the body region of the portrait, the background region, and the field of view edge region and the regular constraint term used for the global image.
(1) The constraint term corresponding to the head region of the portrait is used to constrain a target image matrix corresponding to the head region in the second corrected image to approximate an image matrix obtained after geometric transformation is performed on the first image matrix (namely, M1), to correct stretching deformation occurring in the head region. The geometric transformation in this embodiment represents spatial mapping, that is, an image matrix is mapped into another image matrix through transformation. The geometric transformation in this embodiment may be at least one of image translation transformation (Translation), image scaling transformation (Scale), and image rotation transformation (Rotation).
For ease of description, the constraint term corresponding to the head region of the portrait may be referred to as a head constraint term. When there are a plurality of portraits in the image, head regions of different portraits may correspond to different head constraint terms. For example, for the abnormal-shaped image shown in
The head constraint term may be denoted as Term1, and a formula expression of Term1 is as follows:
Term1(i,j)=SUM(i,j)∈HeadRegionk∥M0(i,j)+Dt(i,j)−Func1k[M1(i,j)]∥.
For an image matrix M0(i, j) of a pixel located in the head region (in other words, (i, j)∈HeadRegionk), a coordinate value M1(i, j) obtained after facial conformal is [u1(i, j),v1(i, j)]T. Herein, Dt(i, j) represents a displacement matrix corresponding to M0(i, j), k represents a kth head region in the image, Func1k represents a geometric transformation function corresponding to the kth head region, and ∥ . . . ∥ represents a vector 2 norm.
It needs to be ensured, by using the head constraint term Term1(i, j), that under the action of the displacement matrix Dt(i, j), the image matrix M0(i, j) tends to be obtained by performing appropriate geometric transformation on M1(i, j). The geometric transformation includes at least one transformation operation of image rotation, image translation, and image scaling.
The geometric transformation function Func1k corresponding to the kth head region represents that all pixels in the kth head region share a same geometric transformation function Func1k, and different head regions correspond to different geometric transformation functions. The geometric transformation function Func1k may be specifically expressed as follows:
Herein, ρ1k represents a scaling coefficient of the kth head region, θ1k represents a rotation angle of the kth head region, and TX1k and TY1k respectively represent horizontal displacement and vertical displacement of the kth head region.
Term1(i, j) may be specifically expressed as follows:
Herein, du(i, j) and dv(i, j) are unknowns that need to be solved. When a constraint equation is subsequently solved, it needs to be ensured that this term is as small as possible.
(2) The constraint term corresponding to the body region is used to constrain a target image matrix corresponding to the body region in the second corrected image to approximate an image matrix obtained after geometric transformation is performed on the second image matrix (namely, M2), to correct stretching deformation occurring in the body region. The geometric transformation in this embodiment may be at least one of image translation transformation, image scaling transformation, and image rotation transformation.
For ease of description, the constraint term corresponding to the body region of the portrait may be referred to as a body constraint term. When there are a plurality of portraits in the image, body regions of different portraits may correspond to different body constraint terms. For example, for the abnormal-shaped image shown in
The body constraint term may be denoted as Term2, and a formula expression of Term2 is as follows:
Term2(i,j)=SUM(i,j)∈BodyRegionk∥M0(i,j)Dt(i,j)−Func2k[M2(i,j)]∥.
For a pixel M0(i, j) located in the head region (in other words, (i, j)∈BodyRegionk), a coordinate value M2(i, j) obtained after body conformal is [u2(i, j), v2(i, j)]T. Herein, Dt(i, j) represents a displacement matrix corresponding to M0(i, j), k represents a kth body region in the image, Func2k represents a geometric transformation function corresponding to the kth body region, and ∥ . . . ∥ represents a vector 2 norm.
It needs to be ensured, by using the body constraint term, that under the action of the displacement matrix Dt(i, j), the image matrix M0(i, j) tends to be obtained by performing appropriate geometric transformation on M2(i, j). The geometric transformation includes at least one transformation operation of image rotation, image translation, and image scaling.
The geometric transformation function Func2k corresponding to the kth body region represents that all pixels in the kth body region share a same geometric transformation function Func2k, and different body regions correspond to different geometric transformation functions. The geometric transformation function Func2k may be expressed as follows:
Herein, ρ2k represents a scaling coefficient of the kth body region, θ2k represents a rotation angle of the kth body region, and TX2k and TY2k respectively represent horizontal displacement and vertical displacement of the kth body region.
Term2(i, j) may be specifically expressed as follows:
Herein, du(i,j) and dv(i,j) are unknowns that need to be solved. When the constraint equation is subsequently solved, it needs to be ensured that this term is as small as possible.
(3) The constraint term corresponding to the field of view edge region is used to constrain a pixel in an initial image matrix corresponding to the field of view edge region in the first corrected image to be displaced along an edge of the first corrected image or towards an outer side of the first corrected image, to maintain or expand the field of view edge region.
For ease of description, the constraint term corresponding to the field of view edge region may be referred to as a field of view edge constraint term. The field of view edge constraint term may be denoted as Term3, and a formula expression of Term3 is as follows:
Term3(i,j)=SUM(i,j)∈EdgeRegion∥M0(i,j)+Dt(i,j)−Func3(i,j)[M0(i,j)]∥.
Herein, M0(i, j) represents image coordinates of a pixel located in the field of view edge region (in other words, (i, j)∈EdgeRegion), Dt(i, j) represents a displacement matrix corresponding to M0(i, j), Func3i,j) represents a displacement function of M0(i, j), and ∥ . . . ∥ represents a vector 2 norm.
It needs to be ensured, by using the field of view edge constraint term, that under the action of the displacement matrix Dt(i, j), the image matrix M0(i, j) tends to be obtained after the coordinate value M0(i, j) is approximately displaced. A displacement rule is to approximately move the pixel only along the edge region or towards the outer side of the edge region, and avoid moving the pixel towards an inner side of the edge region, so that an image information loss caused by subsequent cropping into a rectangular shape can be minimized, and image content in the field of view edge region can even be expanded.
As shown in
Herein, there is a limitation that α(u0, v0) needs to be not less than 0, to ensure that the pixel is not displaced towards the inner side of the field of view edge region, a symbol of β(u0, v0) does not need to be limited, and α(u0, v0) and β(u0, v0) are intermediate unknowns that do not need to be explicitly solved.
Term3(i, j) may be specifically expressed as follows:
Herein, du(i,j) and dv(i,j) are unknowns that need to be solved. When the constraint equation is subsequently solved, it needs to be ensured that this term is as small as possible.
(4) The constraint term corresponding to the background region is used to constrain a pixel in an image matrix corresponding to the background region in the first corrected image to be displaced and a first vector corresponding to the pixel before displacement and a second vector corresponding to the pixel after displacement to be parallel to each other as much as possible, so that the image content in the background region is smooth and continuous, and the image content throughout the portrait in the background region is continuous and consistent in human vision. The first vector represents a vector between the pixel before displacement and a neighboring pixel corresponding to the pixel before displacement, and the second vector represents a vector between the pixel after displacement and a neighboring pixel corresponding to the pixel after displacement.
For ease of description, the constraint term corresponding to the background region may be referred to as a background constraint term. The background constraint term may be denoted as Term4, and a formula expression of Term4 is as follows:
Term4(i,j)=SUM(i,j)∈BkgRegion{Func4(i,j)(M0(i,j),M0(i,j)+Dt(i,j))}.
Herein, M0(i, j) represents image coordinates of a pixel located in the background region (in other words, (i, j)∈BkgRegion), Dt(i, j) represents a displacement matrix corresponding to M0(i, j), Func4(i,j) represents a displacement function of M0(i, j), and ∥ . . . ∥ represents a vector 2 norm.
It needs to be ensured, by using the background constraint term, that under the action of the displacement matrix Dt(i, j), the coordinate value M0(i, j) tends to be obtained after the coordinate value M0(i, j) is approximately displaced. In this embodiment of this application, pixels in the background region may be grouped into different control regions. A size, shape, and quantity of control regions are not limited in this application. In particular, a control region of a background pixel located at a boundary between the foreground portrait and the background needs to extend across the foreground portrait to the other end of the foreground portrait. As shown in
Herein, angle[ ] represents an angle between two vectors, vec1 represents a vector that includes a background pixel [i, j]T before correction and a point in a control region of the pixel, vec2 represents a vector that includes the background pixel [i, j]T after correction and a corrected pixel in the control region of the pixel, and SUM(i+di,j+dj)∈CtrlRegion represents summation of angles between all vectors in the control region.
Term4(i, j) may be specifically expressed as follows:
Term4(i,j)=SUM(i,j)∈BkgRegion{SUM(i+di,j+dj)∈CtrlRegion{angle[vec1,vec2]}}.
Herein, du(i,j) and dv(i,j) are unknowns that need to be solved. When the constraint equation is subsequently solved, it needs to be ensured that this term is as small as possible.
(5) The regular constraint term is used to constrain a difference between displacement matrices of any two adjacent pixels in displacement matrices respectively corresponding to the background region, the head region of the portrait, the body region of the portrait, and the field of view edge region in the second corrected image is less than a preset threshold, so that global image content of the second corrected image is smooth and continuous.
The regular constraint term may be denoted as Term5, and a formula expression of Term5 is as follows:
Term5(i,j)=SUM(i,j)∈AllRegion{Func5(i,j)(Dt(i,j))}.
For pixels M0(i, j) in an entire image range (in other words, (i, j)∈AllRegion), it needs to be ensured, by using the regular constraint term, that displacement matrices Dt(i, j) of adjacent pixels are smooth and continuous, to avoid a great local change. A constraint principle is that a difference between displacement at a pixel [i, j]T and displacement at a neighboring pixel (i+di, j+dj) should be as small as possible (in other words, less than a specific threshold). In this case, Func5(i,j) may be specifically expressed as follows:
Term5(i,j) may be specifically expressed as follows:
Herein, du(i,j) and dv(i,j) are unknowns that need to be solved. When the constraint equation is subsequently solved, it needs to be ensured that this term is as small as possible.
S506. The terminal obtains displacement matrices respectively corresponding to the regions based on the constraint terms and weight coefficients corresponding to the constraint terms.
Specifically, the terminal may set the weight coefficients for the constraint terms of the head region and the body region of the portrait, the field of view edge region, and the background region and the regular constraint term, establish a constraint equation based on the constraint terms and the corresponding weight coefficients, and solve the constraint equation, to obtain an offset of each position point in each region.
It is assumed that a coordinate matrix (which may also be referred to as a target image matrix) of an image (namely, the second corrected image) obtained after algorithm constraint correction is Mt(i, j), where Mt(i, j)=[ut(i, j), vt(i, j)]T. A displacement matrix of the second corrected image with respect to the image matrix M0(i, j) is Dt(i, j), where Dt(i, j)=[du(i, j), dv(i, j)]T. In other words,
Mt(i,j)=M0(i,j)+Dt(i,j);
ut(i,j)=u0(i,j)+du(i,j); and
vt(i,j)=v0(i,j)+dv(i,j).
The weight coefficients are assigned to the constraint terms, and the following constraint equation is constructed:
Dt(i,j)=(du(i,j),dv(i,j))=argmin(α1(i,j)*Term1(i,j)+α2(i,j)*Term2(i,j)+α3(i,j)*Term3(i,j)+α4(i,j)*Term4(i,j)+α5(i,j)*Term5(i,j)).
Herein, α1 (i, j) to α5(i, j) are the weight coefficients (weight matrices) respectively corresponding to Term1 to Term5.
A displacement matrix Dt(i, j) of each pixel in the image is finally obtained by solving the constraint equation by using a least square method, a gradient descent method, or various improved algorithms.
In an embodiment, each of α1 (i, j) to α5(i, j) may be set to a fixed weight value.
In still another embodiment, a correction process is equivalent to selecting a balance point in foreground and background correction, and therefore the weight coefficient may be adaptively generated based on the image content, to implement local weight control. A principle of adaptively configuring the weight coefficient may include but is not limited to at least one of the following:
An image area occupied by the head region of the portrait in the first corrected image is configured to be positively correlated with the weight coefficient corresponding to the head constraint term; and an image area occupied by the body region of the portrait in the first corrected image is configured to be positively correlated with the weight coefficient corresponding to the body constraint term.
That is, when the head region or the body region occupies a large image area, a weight value of the head constraint term or the body constraint term of the corresponding region may be increased because a portrait with a large area is more sensitive to deformation.
A field of view of the head region in the first corrected image is configured to be positively correlated with the weight coefficient corresponding to the head constraint term; and a field of view of the body region in the first corrected image is configured to be positively correlated with the weight coefficient corresponding to the body constraint term.
That is, when the head region or the body region is a region with a large field of view, the weight value of the head constraint term or the body constraint term of the corresponding region may be increased because a portrait with a large field of view tends to be more severely deformed.
Texture significance of the background region in the first corrected image is positively correlated with the weight coefficient corresponding to the background constraint term.
That is, when there are weak texture details for the background region, a weight of the background constraint term of the corresponding region may be decreased because the background region is insensitive to distortion or a fault caused by correction, to better correct the foreground portrait.
A weight coefficient corresponding to the field of view edge constraint term when there is portrait content in the field of view edge region in the first corrected image is configured to be greater than a weight coefficient corresponding to the field of view edge constraint term when there is no portrait content in the field of view edge region.
That is, when there is a portrait on a field of view edge of the image, a weight value of the edge constraint term of the corresponding region may be increased because boundary constraint may affect a correction effect of the foreground portrait, to better correct the foreground portrait.
S507. The terminal obtains the second corrected image through color mapping based on the initial image matrices and the displacement matrices respectively corresponding to the regions.
In specific implementation, after the displacement matrix Dt(i, j) of each pixel in the image is obtained, the terminal obtains a target image matrix Mt(i, j) of the pixel in each region in the first corrected image based on the image matrix M0(i, j) and the displacement matrix Dt(i, j) corresponding to the pixel in each region in the first corrected image, where Mt(i, j)=M0(i, j)+Dt(i, j). Then, an image of target resolution (it is assumed as Wt*Ht) is generated based on the target image matrix Mt(i, j) and a color matrix C(i, j), that is, the second corrected image (namely, a new abnormal-shaped image of Wt*Ht) is obtained. Specifically, normalized Mt(i, j)=[ut(i, j), vt(i, j)]T may be converted into actual image coordinates [it, jt]T. A conversion rule is as follows:
it=Ht/2−vt*Ht; and
jt=ut*Wt+Wt/2.
After the actual image coordinates [it, jt]T are obtained, the color matrix C(i, j) of the pixel [i, j]T in the first corrected image may be supplemented to the pixel [it, jt]T in the second corrected image. Because [it, jt]T may be a decimal, interpolation rounding further needs to be performed on [it, jt]T in some implementations. Commonly used interpolation rounding algorithms include a linear interpolation algorithm, a Lanczos interpolation algorithm, and the like. This is not limited in this application.
S508. The terminal crops the second corrected image into a rectangular image, and outputs the rectangular image.
In this embodiment, the second corrected image obtained in step 507 is an abnormal-shaped image (a non-rectangular image) of Wt*Ht. Therefore, cropping and scaling processing further need to be performed on the abnormal-shaped image, and a rectangular image of the target resolution is output. In an example, as shown in
It may be learned that in this embodiment of this application, the terminal may perform, by region (including the head region, the body region, the background region, and the field of view edge region), algorithm constraint correction processing on the abnormal-shaped image obtained after optical distortion correction, separately perform shape correction on the head region and the body region of the portrait based on different projection algorithms, and separately set independent constraint terms to optimize a solution, to ensure that the portrait is best corrected. For the background region, a position relationship between a background pixel and a pixel in a control region of the background pixel before and after transformation is constrained, to ensure that no background distortion or discontinuity occurs in the image content in the background region due to foreground correction. For the field of view edge region, content adaptive boundary control is used. In this way, there is no problem of a field of view loss, and a field of view boundary can even be expanded, to minimize a content loss caused by cropping and the problem of a field of view loss. Therefore, implementing this embodiment of this application can ensure that after the image is corrected, both the portrait and the background content are natural, continuous, and harmonious, and fit a human visual habit. In this way, user experience is greatly improved, and there are great commercial application benefits.
To better understand the foregoing solution, the following describes the solution with reference to an actual application scenario.
Referring to
S601. The terminal collects a source image.
S602. The terminal performs optical distortion correction on the source image, to obtain an abnormal-shaped image.
For content of S601 and S602, refer to the description of S501 and S502 in the embodiment in
S603. The terminal crops the abnormal-shaped image, to obtain a first corrected image.
Specifically, in this embodiment, the abnormal-shaped image obtained in step 602 is an abnormal-shaped image (non-rectangular image) of Wt*Ht. The abnormal-shaped image may be cropped along an edge position to obtain a rectangular region, and the rectangular region is enlarged to resolution of Wt*Ht, to obtain a rectangular image of Wt*Ht, namely, the first corrected image.
In an example, for specific implementation of cropping and enlargement processing in this step, refer to the description of the embodiment in
S604. The terminal recognizes that the first corrected image includes a portrait region in which stretching deformation occurs, a background region, and a field of view edge region. The portrait region in which stretching deformation occurs includes at least a head region in which stretching deformation occurs and a body region in which stretching deformation occurs.
S605. The terminal determines initial image matrices respectively corresponding to the head region, the body region, the background region, and the field of view edge region.
For specific implementation content of S604 and S605, refer to the description of S503 and S504 in the embodiment in
S606. The terminal constructs constraint terms respectively corresponding to the head region, the body region, the background region, and the field of view edge region and a regular constraint term.
In this step, for the head constraint term, the body constraint term, the background constraint term, and the regular constraint term, refer to related description of S505 in the embodiment in
However, in this step, the targeted first corrected image is a rectangular image, and the field of view edge region of the first corrected image is a rectangular edge region. Therefore, an implementation process of the field of view edge constraint term in this step is different from the implementation process of the field of view edge constraint term in S505 in the embodiment in
In this embodiment, the field of view edge constraint term is used to constrain a pixel in the initial image matrix corresponding to the field of view edge region in the first corrected image is displaced along an edge of the first corrected image, to maintain the field of view edge region.
If the field of view edge constraint term is denoted as Term3, a formula expression of Term3 herein is as follows:
Term3(i,j)=SUM(i,j)∈EdgeRegion∥M0(i,j)+Dt(i,j)−Func3(i,j)[M0(i,j)]∥.
Herein, M0(i, j) represents image coordinates of a pixel located in the field of view edge region (in other words, (i, j)∈EdgeRegion), Dt(i, j) represents a displacement matrix corresponding to M0(i, j), Func3(i,j) represents a displacement function of M0(i, j), and ∥ . . . ∥ represents a vector 2 norm.
It needs to be ensured, by using the field of view edge constraint term, that under the action of the displacement matrix Dt(i, j), the image matrix M0(i, j) tends to be obtained after the coordinate value M0(i, j) is approximately displaced. A displacement rule is to move the pixel only along an edge region of the image, and avoid moving the pixel towards an inner side or an outer side of the image, to minimize an image information loss caused by subsequent cropping into a rectangular shape.
As shown in
A symbol of β(u0, v0) does not need to be limited, to ensure that the pixel can be moved along the boundary. Term3 may be expressed as follows:
Herein, du(i, j) and dv(i, j) are unknowns that need to be solved. When a constraint equation is subsequently solved, it needs to be ensured that this term is as small as possible.
S607. The terminal obtains displacement matrices respectively corresponding to the regions based on the constraint terms and weight coefficients corresponding to the constraint terms.
S608. The terminal obtains a second corrected image through color mapping based on the initial image matrices and the displacement matrices respectively corresponding to the regions, and outputs the second corrected image.
For content of S607 and S608, refer to the description of S506 and S507 in the embodiment in
It may be learned that in this embodiment of this application, the terminal performs optical distortion correction, to obtain an abnormal-shaped image, crops the abnormal-shaped image into a rectangular image, performs algorithm constraint correction processing on the rectangular image by region (including the head region, the body region, the background region, and the field of view edge region), performs shape correction on the head region and the body region of a portrait based on different projection algorithms, and separately sets independent constraint terms to optimize a solution, to ensure that the portrait is best corrected. For the background region, a position relationship between a background pixel and a pixel in a control region of the background pixel before and after transformation is constrained, to ensure that no background distortion or discontinuity occurs in image content in the background region due to foreground correction. For the field of view edge region, content adaptive boundary control is used, to minimize a content loss caused by cropping and a problem of a field of view loss. Therefore, implementing this embodiment of this application can ensure that after the image is corrected, both the portrait and background content are natural, continuous, and harmonious, and fit a human visual habit. In this way, user experience is greatly improved, and there are great commercial application benefits.
To better understand the foregoing solution, the following describes the solution with reference to an actual application scenario.
Referring to
It should be noted that the application scenarios shown in
For example, referring to
The foregoing describes a related system architecture and method in the embodiments of the present invention. Based on a same invention idea, the following describes a related apparatus/device in the embodiments of this application.
The optical distortion correction module 701 is configured to perform optical distortion correction on a collected source image, to obtain a first corrected image. The first corrected image includes a portrait in which stretching deformation occurs and a background region. The portrait in which stretching deformation occurs includes at least a first human body region in which stretching deformation occurs and a second human body region in which stretching deformation occurs.
The algorithm constraint correction module 702 is configured to separately process the first human body region, the second human body region, and the background region, to obtain a second corrected image. The first human body region and the second human body region of the portrait in the second corrected image are based on different algorithms to present different deformation amounts, so that the stretching deformation in both the first human body region and the second human body region is corrected, and image content throughout the portrait in the background region in the first corrected image includes no fault in human vision.
The optical distortion correction module 701 and the algorithm constraint correction module 702 may be specifically configured to implement related method steps of the terminal shown in the embodiment in
In the foregoing embodiments, descriptions of the embodiments have respective focuses. For a part that is not described in detail in an embodiment, refer to related descriptions in other embodiments.
In the several embodiments provided in this application, it should be understood that the disclosed apparatuses may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, division into units is merely division into logical functions and may be other division in an actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in an electrical form or another form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one location, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.
In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in a form of hardware, or may be implemented in a form of a software functional unit.
When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable memory. Based on such an understanding, the technical solutions of this application essentially, or the part contributing to the prior art, or all or some of the technical solutions may be implemented in the form of a software product. The software product is stored in a memory and includes several instructions for instructing a computer device (which may be a personal computer, a server, or a network device) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing memory includes any medium that can store program code, for example, a USB flash drive, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a removable hard disk, a magnetic disk, or an optical disc.
A person of ordinary skill in the art may understand that all or some of the steps of the methods in the embodiments may be implemented by a program instructing related hardware. The program may be stored in a computer-readable memory. The memory may include a flash memory, a read-only memory (English: Read-Only Memory, ROM for short), a random access memory (English: Random Access Memory, RAM for short), a magnetic disk, an optical disc, or the like.
The embodiments of this application are described in detail above. The principles and implementation of this application are described herein through specific examples. The description about the embodiments of this application is merely provided to help understand the method and core ideas of this application. In addition, a person of ordinary skill in the art can make variations and modifications to this application in terms of the specific implementations and application scopes based on the ideas of this application. Therefore, the content of this specification shall not be construed as a limit to this application.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201910896549.6 | Sep 2019 | CN | national |
This application is a continuation of international application PCT/CN2020/114400, filed on Sep. 10, 2020, which claims priority to Chinese Patent Application No. 201910896549.6, filed on Sep. 18, 2019. The disclosures of the aforementioned applications are hereby incorporated by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2020/114400 | Sep 2020 | US |
| Child | 17698650 | US |
