Patent Application
20130163861
The disclosed invention is in the field of image processing and more specifically in the capture, processing and storage of array images. Array image are digital representation of visual data that are organised in frames of picture-elements, or pixels. Each pixel has a digital value that corresponds to one or more channels of information related to the visual content of the location of the said pixel. In many—but not all—cases, the images are generated by an image sensor, an integrated CCD or CMOS device which is read through a pixel-flow decoding circuit. Such circuits have been proposed in document like U.S. Pat. No. 7,391,437.
As proposed in documents such as U.S. Pat. No. 3,971,065 the array can consist of pixels each having information about a different color channel. The missing color information for a location can be inferred by the color channel values of its neighbours through a procedure known as de-mosaicing.
Architectures that process array images have been used since the first digital images were captures by CCD sensors. Most of these architectures process the pixels via a pipeline, i.e. a structure that performs one operation on a pixel while the next step performs another operation on the previous pixel. This method increases the throughput and requires smaller temporary memory structures. Implementations of this approach have been proposed since 1988 in U.S. Pat. No. 5,046,190, and Patent Application No. US 2003/0007703. Other documents that contain descriptions of an image processing pipeline include U.S. Pat. No. 5,852,742 and U.S. Pat. No. 5,915,079. This architecture solves in a satisfactory manner the problem of receiving and preprocessing images from array sensors, specifically when the following steps are image enhancement or image and video compression.
Another class of applications in the field of image processing is that of visual perception, where image data is analyzed in order to detect objects or shapes. Traditional image processing pipelines are not a big help in the acceleration of such algorithms, although many resources are also spent on processing of pixel flows. Two major data structuring methods applied on pixel flows that are used for object and shape detection are Connected Components and Integral Images.
Connected Components (see
Integral Images (see
The current state of the art in image processing supports pixel-flow handling, acceleration for algorithms that detect object and shapes as well as preprocessing at the pixel level of the images. However, all proposed solutions do not take advantage of some generic characteristics that will allow building a more efficient and reusable image processing unit, as the one disclosed with this application.
The characteristic of an image that is presented to the processing medium in the form of a continuous pixel flow is that the flow itself contains all structural information like resolution and color channels. It is therefore straightforward to connect many processing steps that can be applied on the pixel flow, each step generating a modified image as in the case of the processing pipelines, generating a new type of image as in the case of integral images or generating labels for each pixels as in the case of connected component labeling.
The proposed apparatus takes advantage of this characteristic and provides a module which can be used one or more times in a system for accelerating all mentioned functions directly on the pixel flow. Furthermore, the proposed apparatus is capable of generating pixel flows from images stored in memory.
The disclosed pixel flow processing apparatus is a self-contained unit that can be used to receive image streams, to generated image streams, to read images stored in bus-attached memory and to store image streams in bus-attached memory. The said pixel flow processor prepares images for further analysis by other units or programs.
The disclosed pixel flow processor consists of a number of pixel flow input units, a number of pixel flow output units, an image processing pipeline, a connected component labeling unit and an integral image calculation unit. Two or more of the disclosed pixel flow processors can be used in the same system, connected through a shared bus, through direct pixel flow channels or using a combination of both of the said connection types.
In one embodiment, the disclosed apparatus will be able to run, without relying on other resources, the following operations:
The disclosed apparatus is preferably—but not exclusively—used as part of a system-on-chip, for acceleration of functions related to processing of image streams.
The example diagram of
In the example of
In the example of
In the example of
In the example of
In the example of
In the example of
In order to display images processed by the said example system of
The disclosed apparatus is hereby described by reference to the block diagram of
The input for all processing is performed by one or more pixel flow readout units 210. For a complete implementation of the disclosed apparatus, at least two pixel flow readout units are required, one attached to the shared bus 190 and another that reads continuous flows 294. The readout unit synchronises to frame and line boundaries 530, 540 of the image and outputs a continuous pixel flow where a data-valid 520 signal is used to denote that at a current clock 510 tick the flow does not contain a pixel 550. One implementation of this flow can be as shown in
The output of all processing is generated by one or more pixel flow output units 260. For a complete implementation of the disclosed apparatus, at least two pixel flow output units are required, one attached to the shared bus 190 and another that generates continuous flows 296. The said output unit reads a pixel flow equivalent to the one shown in
The pixel flow input 210 and output 260 units can read and write images from bus-attached devices. To do so, they are connected to a shared bus 190 via an image reading bus interface unit 272 and an image writing bus interface unit 274. The bus interface units are bus master devices that access other devices, like 140 memories, via read and write bus access cycles. The bus interface device that reads 272 will fetch an entire image region by a sequence of bus read commands and give it in the form of a pixel flow to the readout unit 210. The read sequences are be constructed by single reads and by a number of multiple reads, i.e. burst reads. The bus interface unit that writes 274 will get a continuous pixel flow as generated by the output unit 260 and write it via a sequence of bus write commands to memory or other bus-attached device as a complete image region. The write sequences are constructed by single writes and by a number of multiple writes, i.e. burst writes.
The pixel flow provided by the input 210 unit and by the processing pipeline 300 is connected to a unit 230 that calculates and generates in the form of a pixel flow the integral image. An integral image, also known as a summed area table, is defined as shown in
The pixel flow provided by the input 210 unit and by the processing pipeline 300 is connected to a unit 250 that calculates and generates in the form of a pixel flow an image with labeled pixels, each label corresponding to an arbitrary connected region in the original image.
The pixel flow received by the input 210 units is provided as input to a processing pipeline block 300. The said processing pipeline consists of means to modify the pixel values in order to prepare them for further processing. It is structured as a data-flow machine with processing steps 320, 330, 340, 350, 360.
The first processing step of the said processing pipeline is pixel pattern translation 320. This step reads the pixel flow 312 and translates it into standard pixel flow with each pixel having all color channels. As shown in
The second processing step is color space conversion 330. This step reads pixels as prepared by the pattern translation step 320 and converts the values of their color channels into a different color space. It contains means for support for sub-sampling of selected color channels. In one embodiment shown in
The third step is image scaling 340. This step reads the image and outputs a new pixel flow corresponding to an image with different dimensions. In one embodiment shown in
The fourth step 350 is primary intended to be used for applying color correction operations like white balance on the input images. It is a rather straightforward block that rescales each color component of each pixel by a factor which is externally provided, either by software or by an automatic white balance estimator.
The fifth step 360, feature extraction, does not alter the color channels of each pixel, but rather reads them in order to calculate other pixel features that may be used in further processing. In one embodiment shown in
Before exiting the processing pipeline, a structure 370, based on multiplexers selects which channels are mapped to which channels of the output pixel flow 314. This allows for creating configurable pixel structures, as best suited for each specific image processing algorithm.
DISCLAIMER: The work that led to the development of this invention, was co-financed by Hellenic Funds and by the European Regional Development Fund (ERDF) under the Hellenic National Strategic Reference Framework (NSRF) 2007-2013, according to Contract no. MICRO2-09 of the Project “SC80” within the Programme “Hellenic Technology Clusters in Microelectronics—Phase-2 Aid Measure”.
