The present disclosure is generally related to image processing and, more particularly, to utilization of a resizer in image processing.
Unless otherwise indicated herein, approaches described in this section are not prior art to the claims listed below and are not admitted as prior art by inclusion in this section.
As the pixel count of image sensors continues to increase, the performance of image signal processor (ISP) modules need to be boosted to provide zero-shutter delay image-capture capability with real-time preview or video recording. Although more powerful processing units could be utilized as a straightforward solution, such approach would result in a sizable increase in cost. Another approach to providing high throughout would be to down-scale the previous and video image in the ISP to effectively reduce overall throughput, but user experience would be negatively impacted.
The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce concepts, highlights, benefits and advantages of the novel and non-obvious techniques described herein. Select implementations are further described below in the detailed description. Thus, the following summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.
An objective of the present disclosure is to propose schemes, solutions, concepts, designs, methods and apparatuses for image processing with a frontal resizer that resizes, or down-scales, image data of a received image before further processing, including demosaicing, is performed on the image data. It is believed that proposed schemes in accordance with the present disclosure may avoid high cost and reduction in overall throughput associated with existing approaches.
In one aspect, a method may involve receiving image data of a captured image from an image sensor. The method may also involve processing the image data through a pipeline in which the image data is resized before further processing is performed to provide processed data of a processed image used in preview or video recording.
In another aspect, a method may involve receiving image data of a captured image from an image sensor. The method may also involve resizing the image data by a resizer to provide resized data. The method may further involve demosaicing the resized data by a demosaicing module (DM) before further processing to provide processed data of a processed image used in preview or video recording.
In yet another aspect, an apparatus may include at least one image sensor configured to capture an image to provide image data. The apparatus may also include a processor coupled to receive the image data from the image sensor. The processor may include an image signal processing (ISP) pipeline through which the image data is processed to provide processed data of a processed image used in preview or video recording such that the image data is resized before further processing is performed in the ISP pipeline to provide the processed data.
It is noteworthy that, although description provided herein may be in the context of certain topologies such as a resizer in an ISP, the proposed concepts, schemes and any variation(s)/derivative(s) thereof may be implemented in, for and by other image processing technologies that are to be developed in the future as well as any other image processing technology not mentioned herein whether implicitly or explicitly. Thus, the scope of the present disclosure is not limited to the examples described herein.
The accompanying drawings are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of the present disclosure. The drawings illustrate implementations of the disclosure and, together with the description, serve to explain the principles of the disclosure. It is appreciable that the drawings are not necessarily in scale as some components may be shown to be out of proportion than the size in actual implementation in order to clearly illustrate the concept of the present disclosure.
Detailed embodiments and implementations of the claimed subject matters are disclosed herein. However, it shall be understood that the disclosed embodiments and implementations are merely illustrative of the claimed subject matters which may be embodied in various forms. The present disclosure may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments and implementations set forth herein. Rather, these exemplary embodiments and implementations are provided so that description of the present disclosure is thorough and complete and will fully convey the scope of the present disclosure to those skilled in the art. In the description below, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments and implementations.
Under a proposed scheme in accordance with the present disclosure, ISP 120 may be configured, designed or otherwise built with an image processing pipeline which down-scales the image data before further processing is performed to provide processed data of a processed image. Under the proposed scheme, in processing the image data through the pipeline, ISP 120 may process the image data through the pipeline in which down-scaling by down-scaler 122 is performed before demosaicing by DM 124 to provide the processed data. For instance, in processing the image data through the pipeline, ISP 120 may process the image data sequentially through down-scaler 122, followed by DM 124, and then resizer 126. In some cases, down-scaler 122 may down-scale the image data to provide down-scaled data with a down-scaled pixel count lower than a sensor pixel count of the image data from image sensor(s) 110. Under the proposed scheme, a sensor pixel count of image sensor(s) 110 may be higher than a down-scaled pixel count of the image data after the down-scaling which may be higher than a pixel count of the processed data. In such cases, the pixel count of the processed data may be 8 megapixels in case display 140 includes a 4K2K display or 2 megapixels in case display 140 includes a FHD display.
Under the proposed scheme, down-scaler 122 may be implemented with either a bin (shown in
Under a proposed scheme in accordance with the present disclosure, ISP 220 may be configured, designed or otherwise built with an image processing pipeline which down-scales the image data before further processing is performed to provide processed data of a processed image. Under the proposed scheme, in processing the image data through the pipeline, ISP 220 may process the image data through the pipeline in which down-scaling by fixed-size binning module 222 is performed before demosaicing by DM 124 to provide the processed data. For instance, in processing the image data through the pipeline, ISP 220 may process the image data sequentially through fixed-size binning module 222, followed by DM 124, and then resizer 126. In some cases, fixed-size binning module 222 may reduce the size of image data by a fixed amount (e.g., 50% or 75%) to provide down-scaled data with a fixed-sized pixel count lower than a sensor pixel count of the image data from image sensor(s) 110. Thus, a sensor pixel count of image sensor(s) 110 may be higher than a fixed-sized pixel count of the image data after the down-sizing which may be higher than a pixel count of the processed data. The pixel count of the down-sized data may be denoted as Pb, with Pb=Ps/2 or Ps/4 and Pb>>Pd in case the fixed down-scaling ratio of fixed-size binning module 222 is 1/2 or 1/4, respectively.
However, as fixed-size binning module 222 down-scales the image data by a fixed ratio, the resultant processed data may not be visually pleasant to an end user. Accordingly, the use of fixed-size binning module 222 may be suitable for designs and applications for non-vision use supports such as artificial intelligence (Al) and computer vision (CV).
Under a proposed scheme in accordance with the present disclosure, ISP 320 may be configured, designed or otherwise built with an image processing pipeline which resizes the image data before further processing is performed to provide processed data of a processed image. Under the proposed scheme, in processing the image data through the pipeline, ISP 320 may process the image data through the pipeline in which resizing by resizer 322 is performed before demosaicing by DM 124 to provide the processed data. For instance, in processing the image data through the pipeline, ISP 320 may process the image data sequentially through resizer 322, followed by DM 124, and then resizer 126. In some cases, resizer 322 may resize the image data to provide resized data with a resized pixel count lower than a sensor pixel count of the image data from image sensor(s) 110. Under the proposed scheme, a sensor pixel count of image sensor(s) 110 may be higher than a resized pixel count of the image data after the resizing which may be higher than a pixel count of the processed data.
As resizer 322 may down-scale the image data by an arbitrary or adjustable ratio, the resultant quality may be better. The pixel count of the resized data may be denoted as Pr, with Ps>>Pr>>Pd. With resizer 322 placed before DM 124 in the image processing pipeline, a resultant image with sufficient quality for preview and/or video recording may be produced. Advantageously, this design may save computation resources in the pipeline before DM 124. Moreover, with the design shown in
At 410, process 400 may involve ISP 320 of apparatus 300 receiving image data of a captured image from image sensor(s) 110. Process 400 may proceed from 410 to 420.
At 420, process 400 may involve ISP 320 processing the image data through a pipeline in which the image data is resized (e.g., by resizer 322) before further processing is performed by other components of ISP 320 to provide processed data of a processed image used in preview or video recording on display 140.
In some implementations, in processing the image data through the pipeline, process 400 may involve ISP 320 processing the image data through the pipeline in which resizing by resizer 322 is performed before demosaicing by DM 324 to provide the processed data.
In some implementations, in processing the image data through the pipeline, process 400 may involve ISP 320 processing the image data sequentially through resizer 322 as a first resizer, DM 324, and then resizer 326 as a second resizer. In such cases, resizer 322 may resize the image data to provide resized data with a resized pixel count lower than a sensor pixel count of the image data from image sensor(s) 110.
In some implementations, a sensor pixel count of image sensor(s) 110 may be higher than a resized pixel count of the image data after the resizing which may be higher than a pixel count of the processed data. In such cases, the pixel count of the processed data may be the same as display size such as, for example and without limitation, 8 megapixels (8M) in a 4K2K display, 2 megapixels (2M) in a FHD display, or a different size depending on actual implementations.
In some implementations, image sensor(s) 110 may include a plurality of image sensors. In such cases, in receiving the image data from image sensor(s) 110, process 400 may involve ISP 320 receiving respective image data of a respective captured image from one of the plurality of image sensors each having a respective pixel count. Moreover, in processing the image data through the pipeline, process 400 may involve resizer 322 resizing the respective image data by a respective ratio corresponding to the respective pixel count of the one of the plurality of image sensors.
In some implementations, in processing the image data through the pipeline in which the image data is resized, process 400 may involve resizer 322 resizing the image data by a ratio which is variable based on one or more of: (a) a pixel count of image sensor(s) 110; (b) a field of view (FOV) of the preview; (c) a capacity of ISP 320 in processing the image data; and (d) a power capacity concern of the system (e.g., apparatus 300).
At 510, process 500 may involve ISP 320 receiving image data of a captured image from image sensor(s) 110. Process 500 may proceed from 510 to 520.
At 520, process 500 may involve resizer 322 resizing the image data to provide resized data. Process 500 may proceed from 520 to 530.
At 530, process 500 may involve DM 324 demosaicing the resized data before further processing by other components of ISP 320 to provide processed data of a processed image used in preview or video recording on display 140.
In some implementations, in resizing the image data, process 500 may involve resizer 322 providing the resized data with a resized pixel count lower than a sensor pixel count of the image data from image sensor(s) 110.
In some implementations, a sensor pixel count of image sensor(s) 110 may be higher than a resized pixel count of the resized data which may be higher than a pixel count of the processed data. In such cases, the pixel count of the processed image may be the same as display size such as, for example and without limitation, 8M in a 4K2K display, 2M in a FHD display, or a different size depending on actual implementations.
In some implementations, image sensor(s) 110 may include a plurality of image sensors. In such cases, in receiving the image data from image sensor(s) 110, process 500 may involve ISP 320 receiving respective image data of a respective captured image from one of the plurality of image sensors each having a respective pixel count. Moreover, in processing the image data through the pipeline, process 500 may involve resizer 322 resizing the respective image data by a respective ratio corresponding to the respective pixel count of the one of the plurality of image sensors.
In some implementations, in resizing the image data, process 500 may involve resizer 322 resizing the image data by a ratio which is variable based on one or more of: (a) a pixel count of image sensor(s) 110; (b) a field of view (FOV) of the preview; (c) a capacity of ISP 320 in processing the image data; and (d) a power capacity concern of the system (e.g., apparatus 300).
The herein-described subject matter sometimes illustrates different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are merely examples, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected”, or “operably coupled”, to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable”, to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable and/or physically interacting components and/or wirelessly interactable and/or wirelessly interacting components and/or logically interacting and/or logically interactable components.
Further, with respect to the use of substantially any plural and/or singular terms herein, those having skill in the art can translate from the plural to the singular and/or from the singular to the plural as is appropriate to the context and/or application. The various singular/plural permutations may be expressly set forth herein for sake of clarity.
Moreover, it will be understood by those skilled in the art that, in general, terms used herein, and especially in the appended claims, e.g., bodies of the appended claims, are generally intended as “open” terms, e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc. It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to implementations containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an,” e.g., “a” and/or “an” should be interpreted to mean “at least one” or “one or more;” the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should be interpreted to mean at least the recited number, e.g., the bare recitation of “two recitations,” without other modifiers, means at least two recitations, or two or more recitations. Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention, e.g., “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc. In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention, e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc. It will be further understood by those within the art that virtually any disjunctive word and/or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”
From the foregoing, it will be appreciated that various implementations of the present disclosure have been described herein for purposes of illustration, and that various modifications may be made without departing from the scope and spirit of the present disclosure. Accordingly, the various implementations disclosed herein are not intended to be limiting, with the true scope and spirit being indicated by the following claims.
The present disclosure is part of a non-provisional application claiming the priority benefit of U.S. Provisional Patent Application No. 62/818,787, filed 15 Mar. 2019, the content of which being incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
62818787 | Mar 2019 | US |