Patent Application
20100110213
This application is based upon and claims the benefit of priority from the prior Japanese Patent Application No. 2008-280180, filed on Oct. 30, 2008; the entire contents of which are incorporated herein by reference.
1. Field of the Invention
The present invention relates to an image processing processor, an image processing method, and an imaging apparatus.
2. Description of the Related Art
Conventionally, an image processing processor that is mounted on an imaging apparatus such as a video camera and a digital still camera, and that performs image processing on image data that is output from a complementary metal oxide semiconductor (CMOS) sensor and the like has a configuration in which a buffer to temporarily store image data is used to perform image processing sequentially on image data in a raster-scanned order that is successively input at a fixed data rate, to perform pipeline processing.
Meanwhile, for such an image processing processor, there has been a demand for reduction in the area of an image processing processor by decreasing the capacity of a buffer, because there is severe limitation in a physical area generally. To respond to this demand, it can be considered that synchronization points are increased (shorten a synchronization cycle) to decrease the size of image data that is handled at a single synchronization operation, so that the capacity of the buffer can be reduced.
Japanese Patent Application Laid-Open No. 2003-29979 discloses a technique of reducing the size of communicated image data to a size equal to or smaller than the number of pixels of one line with such a configuration that synchronization is performed when received image data reaches a predetermined data amount. However, although the technique enables reduction of the capacity of an input buffer in which image data that is input from an external device is temporarily stored and the capacity of an output buffer in which image data that is to be output to an external device is temporarily stored, an intermediate buffer requires the capacity at least sufficient for a line or a frame because the intermediate buffer in which a processor core temporarily stores intermediate data is configured to store image data such that a position of a pixel can be controlled by a memory address. Therefore, the reduction of the capacity of a buffer has been limited.
An image processing processor according to an embodiment of the present invention comprises:
an input unit that sequentially writes a digital image signal to be input in a first buffer while counting number of pixels of the digital image signal, and that reads the digital image signal written in the first buffer to write in a second buffer;
a command storage unit that stores an image processing command to which process delay information is added, the process delay information indicating a delay amount required until image processing is started by the image processing command since input of the digital image signal to the input unit;
a command fetching/issuing unit that fetches the image processing command from the command storage unit, that calculates a position of a pixel targeted for the image processing by the fetched image processing command in an image frame of a digital image signal based on the process delay information added to the fetched image processing command and a counter value indicating the number of pixels obtained by the input unit, and that issues the fetched image processing command when a calculated position is in a valid area, and does not issue the fetched image processing command when the calculated position is not in the valid area; and
a command executing unit that performs image processing on the pixel targeted for the image processing written in the second buffer by executing the image processing command issued by the command fetching/issuing unit.
An image processing method according to an embodiment of the present invention comprises:
writing a digital image signal to be input sequentially in a first buffer while counting number of pixels of the digital image signal, and reading the digital image signal written in the first buffer to write in a second buffer;
fetching an image processing command to which process delay information is added from a command storage unit, the process delay information indicating a delay amount required until image processing is started by the image processing command since input of the digital image signal;
calculating a position of a pixel targeted for the image processing by the fetched image processing command in an image frame of a digital image signal based on the process delay information added to the fetched image processing command and a counter value indicating the number of pixels;
issuing the fetched image processing command when a calculated position is in a valid area, and arranging not to issue the fetched image processing command when the calculated position is not in the valid area; and
performing the image processing on the pixel targeted for the image processing written in the second buffer by executing the issued image processing command.
An imaging apparatus according to an embodiment of the present invention comprises:
a digital-image-signal generating unit that outputs a digital image signal at a fixed data rate from an optical image of a subject; and
an image processing processor to which a digital image signal generated by the digital-image-signal generating unit is sequentially input, and that sequentially performs image processing on the digital image signal, in synchronization with the input, wherein
the image processing processor comprises:
an input unit that sequentially writes a digital image signal to be input in a first buffer while counting number of pixels of the digital image signal, and that reads the digital image signal written in the first buffer to write in a second buffer;
a command storage unit that stores an image processing command to which process delay information is added, the process delay information indicating a delay amount required until image processing is started by the image processing command since input of the digital image signal to the input unit;
a command fetching/issuing unit that fetches the image processing command from the command storage unit, that calculates a position of a pixel targeted for the image processing by the fetched image processing command in an image frame of the digital image signal based on the process delay information added to the fetched image processing command and a counter value indicating the number of pixels obtained by the input unit, and that issues the fetched image processing command when a calculated position is in a valid area, and does not issue the fetched image processing command when the calculated position is not in the valid area; and
a command executing unit that performs the image processing on the pixel targeted for the image processing written in the second buffer by executing the image processing command issued by the command fetching/issuing unit.
Exemplary embodiments of an image processing processor, an image processing method, and an imaging apparatus according to the present invention will be explained below in detail with reference to the accompanying drawings. The present invention is not limited to the embodiments.
An image processing processor according to an embodiment of the present invention is mounted on an imaging apparatus, such as a video camera and a digital still camera, that has a digital-image-signal generating unit, such as a CMOS sensor and a charge coupled device (CCD) sensor, that outputs a digital image signal from an optical image of a subject, for example, at a predetermined data rate. To the image processing processor, the digital image signal thus output is sequentially input.
Buffers and a flow of image data are explained. The input unit 2 includes an input buffer 24 that temporarily stores the image data that is input in a raster-scanned order. The processor core 3 includes an intermediate buffer 35 that is used as a work area of image processing. The input unit 2 reads image data from the input buffer 24 and writes in the intermediate buffer 35. The intermediate buffer 35 stores pixel data of a pixel (a target pixel) being a subject of the image processing and adjacent pixels that are used in the image processing to be performed on the target pixel, and intermediate data that is generated as a result of the image processing. The output unit 4 includes an output buffer 41, and reads image data obtained as a result of the image processing from the intermediate buffer 35 and stores in the output buffer 41 temporarily. The image data stored in the output buffer 41 is output to an external device in a raster-scanned order.
The operation of reading image data from the input buffer 24 and writing in the intermediate buffer 35, the image processing performed on the image data written in the intermediate buffer 35, the operation of writing a result of the image processing in the output buffer 41, and the operation of outputting the image data from the output buffer 41 to an external device are performed in synchronization with the input of image data from an external device, at each process-unit pixel amount being the number of pixels of image data that is input in a period from a point of synchronization to a point of next synchronization in a pipeline processing manner. Therefore, the input buffer 24 and the output buffer 41 are configured to have the capacity corresponding to the process-unit pixel amount. For example, the input buffer 24 and the output buffer 41 have the capacity equal to or equivalent to the process-unit pixel amount. To reduce the capacity of the input buffer 24 and the output buffer 41, the cycle of synchronization is set to be short such that the process-unit pixel amount is a minimum value, such as 1 pixel.
Image data to be input from an external device is explained. Image data input from an external device includes an invalid area such as a blanking period.
Referring back to
The processor core 3 includes the intermediate buffer 35, a command memory 31 serving as a command storage unit that stores a command (an image processing command) concerning the image processing, an operation unit 33 and a load/store unit 34 serving as a command executing unit that executes a command, and a command fetching/issuing unit 32 serving as a command fetching/issuing unit that fetches a command from the command memory 31 and issues a command to the operation unit 33 and the load/store unit 34. The load/store unit 34 reads and writes image data from and to the intermediate buffer 35.
A format of a command stored in the command memory 31 is explained. A command (hereinafter, “command string”) stored is the command memory 31 has a structure in which a plurality of commands (hereinafter, “small commands”) that can be executed at the same time are included.
The target-pixel designation command has a process delay information that indicates a delay amount required until the operation of a small command in the command string is started on the first pixel since the first pixel of an image frame to which the target pixel belongs is input from an external device, and the position of the target pixel is calculated based on the process delay information and a counter value of the pixel counter 22. When the target pixel is positioned in a valid area, small commands in the command string are executed, and when the target pixel is positioned in an invalid area, the small commands in the command string are not executed. The delay amount indicated by the process delay information is expressed by the number of input pixels that are input in a period from the point of synchronization to a point of another synchronization, or time such as the number of image clocks corresponding to the number of input pixels.
The small commands to be stored in the first area, the second area, and the third area include synchronization command to synchronize input of image data from an external device and image processing performed on the input image data, and a jump command to change the address from which a next command string is fetched, in addition to a load command to read image data from a reading source, a store command to write image data in a writing destination, an operation command to execute image processing, and a command to execute an operation of pixels.
The command fetching/issuing unit 32 fetches a command string and then issues a small command that is included in the command string. That is, the command fetching/issuing unit 32 issues an operation command to the operation unit 33, and issues a load command and a store command to the load/store unit 34. The command fetching/issuing unit 32 executes a synchronization command, a jump command, and a target-pixel designation command by itself.
The command fetching/issuing unit 32 includes a pixel-position calculating unit 321 that calculates the position of a target pixel based on the process delay information included in the target-pixel designation command and the value of the pixel counter 22, to execute the target-pixel designation command, and a command invalidating unit 322 that determines whether the calculated position of the target pixel is in a valid area based on the value of the valid area register 21, and that arranges not to issue small commands included in the command string to the operation unit 33 and the load/store unit 34 when the position of the target pixel is not in a valid area.
Moreover, the command fetching/issuing unit 32 includes a synchronizing unit 323 that executes a synchronization command. When a synchronization command is included in a fetched command string, the synchronizing unit 323 interprets the synchronization command, and executes a control to cause the command fetching/issuing unit 32 to wait to execute the operation of fetching a command string until a next synchronization timing signal is issued by the synchronization-timing-signal generating unit 23. The synchronizing unit 323 cancels the standby of the operation of fetching a command when a next synchronization timing signal is issued, and causes the command fetching/issuing unit 32 to fetch a next command string.
When a jump command is executed, the command fetching/issuing unit 32 reads a next command string from a reading source in the command memory 31 that is designated by the jump command.
The output unit 4 includes the output buffer 41 as described above.
Next, the operation of the image processing processor 1 according to the first embodiment configured as described above is explained. In the following explanations of the operation, it is assumed that the image frame shown in
The processor core 3 performs the operation of performing various kinds of image processing on the 1 pixel of the target pixel written in the intermediate buffer 35 based on command strings 1 to n-1 in which no synchronization command is included and a command string n in which the synchronization command is included during the period from the synchronization timing signal is detected until a next synchronization timing signal is detected in pipeline. A flowchart shown on the right in
As shown in the flowchart on the right in
As described above, the image data input to the input unit 2 and the image processing on the image data by the processor core 3 are synchronized. As another synchronization method, a polling method in which a register that indicates that accumulated image data has reached the process-unit pixel amount in the input unit 2 and the output unit 4 is provided and the processor core 3 reads the register at certain time intervals to achieve the synchronization, and an interrupt method in which the input unit 2 notifies that accumulated image data has reached the process-unit pixel amount by a hardware signal and a receiver thereof is connected to an interrupt controller of the processor core 3 can be applied. However, the synchronization method according to the first embodiment has less overhead than these synchronization methods, and therefore synchronization can be achieved in a higher speed. That is, compared to the polling method and the interrupt method, it is possible to set the cycle of synchronization short while keeping the amount of increase of overhead low.
A shown in
When the position of a pixel targeted for the operation in the image frame shown in
C−D=(X−1)+(Y−1)×10 (Equation 1)
(where X and Y are positive integers satisfying 1≦X≦10, 1≦Y≦3) is satisfied. Therefore, X and Y can be calculated from the counter value C and the delay amount D. For example, when the current counter value of the pixel counter 22 is 14, and the delay amount is D pixels, it is calculated as (X, Y)=(3, 1). That is, the position of the target pixel is the third pixel on the first line.
Following Step S21, the command invalidating unit 322 determines whether the position of the pixel is in a valid area referring to the valid area register 21 (Step S22). Although the form of expression of the valid area by the valid area register 21 is not particularly limited, because the valid area in the image frame shown in
When the position of the pixel is in the valid area (YES at Step S22), the command fetching/issuing unit 32 issues small commands stored in the first to the third areas (Step S23). When the position of the pixel is not in the valid area (NO at Step S22), the command invalidating unit 322 invalidates the small commands included in the command string, and the execution of the command string is ended without issuing the small commands by the command fetching/issuing unit 32. However, even when the pixel targeted by the command string n is positioned in the invalid area, to wait until next synchronization timing signal, only a synchronization command is issued to the synchronizing unit 323 as an exemption, even when the position of the pixel is not in the valid area.
As described above, the command fetching/issuing unit 32 calculates the position of a pixel targeted by a small command included in a command string based on the process delay information indicating the delay amount in the fetched command string and the counter value. Therefore, the intermediate buffer 35 is freed from such a constraint that a large amount of data in a line unit or a frame unit is required as in the technique disclosed in Japanese Patent Application Laid-Open No. 2003-29979, and the capacity of the intermediate buffer 35 can be reduced. For example, when an operation of image processing is to be performed on a pixel in center using 3 pixels in H direction×3 pixels in V direction, in the technique disclosed in Japanese Patent Application Laid-Open No. 2003-29979, the capacity for at least three lines is required just for an area to store the image data of a target pixel before subjected to the image processing and adjacent pixels used in the operation of the image processing for the target pixel, even though the process-unit pixel amount corresponds to the size smaller than one line. To the contrary, in the image processing processor 1, the capacity can be reduced to the size corresponding to 2 lines+3 pixels if the process-unit pixel amount is set to 1 pixel.
When the calculated position of the pixel is in the invalid area, the operation that the small commands included in the command string are not issued is executed. Therefore, the intermediate buffer 35 does not require the capacity to store intermediate data of the image processing for the pixel in the invalid area. Accordingly, in the intermediate buffer 25, the capacity to store the intermediate data of the invalid area can be further reduced.
If the operation of calculating the position of a target pixel and the operation of determining whether the calculated position is in the valid area are executed by software, several conditional branches are required, and overhead originated in such conditional branches increases. However, because the image processing processor 1 performs these operations by hardware, the operations can be performed in a high speed compared to the case of software.
The input unit 2 sequentially writes image data that is input to the input unit 2 as shown in the first line to the fourth line in the input buffer 24 as shown in the fifth line. The input unit 2 then sequentially writes the written image data into the intermediate buffer 25 one pixel each. The processor core 3 performs a process A shown on the sixth line and a process B shown on the seventh line based on the command strings 1 to n. The output unit 4 reads the image data subjected to the process B one pixel each and writes in the output buffer 41 as shown in the eighth line.
The command string for the operation of the process A includes the target-pixel designation command that includes the process delay information specifying the delay amount of 11 pixels as shown by an arrow (1) in
As described above, according to the first embodiment, the image processing processor is configured such that a command including information indicating a delay amount required until image processing is executed since the input of a digital image signal is fetched, the position of a pixel is calculated based on the delay amount and a counter value of pixels input from an external device, and the fetched command is executed when the position of the pixel is in the valid area while the fetched command is not executed when the position of the pixel is in the invalid area. Therefore, if the synchronization cycle is set to be short, the capacity of the input buffer, the intermediate buffer, and the output buffer can be reduced. Thus, an image processing processor in which the capacity of buffers is minimized can be provided.
Although in the above explanations, the input unit 2, the processor core 3, and the output unit 4 have the input buffer 24, the intermediate buffer 35, and the output buffer 41, respectively, positions at which each buffer is provided is not limited thereto.
Moreover, although the pixel counter 22 is explained that the counter value is reset to indicate “1” when the first pixel of an image frame is input, as an example, it can be configured to be reset in any way, as long as a position of a pixel and a counter value is associated so that the position of a pixel can be calculated from the relation between the counter value and the delay amount. For example, the pixel counter 22 can be configured as a two-dimensional counter constituted by two kinds of counters for H direction and V direction such that the counter value of H direction is incremented each time 1 pixel is input, and the counter value of V direction is incremented each time pixels of one line are input. Similarly, the delay amount can also be expressed by two kinds of amounts in H direction and V direction.
Furthermore, although the expression of the delay amount in the process delay information is not specifically described, the delay amount can be expressed directly by a value indicating the delay amount, or indirectly by indicating a register in which a value of the delay amount is stored.
Moreover, although the method of setting a value indicated by the valid area register 21 is not specifically mentioned, it can be configured to be set at the initial setting, or a function of automatically setting the value based on the horizontal synchronizing signal and the vertical synchronizing signal can be added to the input unit 2, for example. If it is configured to add the function of automatically setting the value, a process of the initial setting of the valid area register 21 can be omitted.
Furthermore, a jump command function can be added to the synchronization command. By such an arrangement, a jump command is not required to be included in a command string in addition to the synchronization command. When the jump command function is added to the synchronization command, it can be arranged to jump to a predetermined address, or to specify an address of a jump destination by specifying an immediate value or a register.
An image processing processor according to a second embodiment of the present invention is explained next.
As shown in
The processor core 7 performs image processing on data of single instruction multiple data (SIMD). Specifically, the processor core 7 includes a SIMD operation unit 71 that performs the same operation on a plurality of pixels at the same time by a single operation command, in place of the operation unit 33. The process-unit pixel amount is set to the number of pixels for which the SIMD operation unit 71 can perform an operation, that is a SIMD width.
The input unit 6 includes a synchronization-timing-signal generating unit 61 in place of the synchronization-timing-signal generating unit 23. The synchronization-timing-signal generating unit 61 issues a synchronization timing signal each time the image data corresponding to the process-unit pixel amount, that is the SIMD width, is input. When the number of horizontal pixels is not an integral multiple of the SIMD width, the synchronization-timing-signal generating unit 61 partially adjusts an interval of issuing the synchronization timing signal to bring a synchronization point to the head of a following line so that image data of different lines are not included in the same SIMD data.
A flow of image data in the image processing processor 5 configured as described above is explained next.
As shown in
As described above, according to the second embodiment, the same operation can be performed on pixels of the process-unit pixel amount. Therefore, as compared to the first embodiment, the operation of the command fetching/issuing unit is facilitated.
Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2008-280180 | Oct 2008 | JP | national |
