Patent Grant
8531546
1. Field of the Invention
The present invention relates to an image processing apparatus.
2. Description of the Background Art
In response to a recent trend towards higher pixel resolution of a digital camera, higher transmission speed and higher image processing speed of captured image data have been required.
With reference to
The interlace sensor 2 outputs the captured image of one frame in two fields, namely, odd and even fields. However, image processing using the SPU 6 and later units requires peripheral pixel information, and hence, information only about the odd field or about the even field is not responsive to the image processing. In response, subsequent process steps are put on hold until the interlace sensor 2 outputs the image data of two fields to reproduce the image data of one frame as original image data 16 on the main memory 5.
After the image data of one frame is reproduced as the original image data 16, a real-time processing unit (hereinafter referred to as “RPU”) 7 performs signal processing such as interpolation, color space conversion and pseudo color suppression on the image data written to a line memory 8 from the main memory 5. The image data processed by the RPU 7 is thereafter sent to an image compression and expansion unit 13 through line buffers A24 and B25 on the main memory 5. For compression into the JPEG (Joint Photographic Experts Group) format, the line buffers A and B each normally have a capacity of 8 lines.
More specifically, after the processed image data is sent to the line buffer A24, the RPU 7 continues to process subsequent image data read from the original image data 16, which processed data is then sent to the line buffer B25. The image data sent in advance to the line buffer A24 is concurrently sent to the image compression and expansion unit 13 and is subjected to processing therein. The RPU 7 continues to be operative to send subsequent image data to the empty line buffer A24, concurrently with which the image data sent in advance to the line buffer B25 is sent to the image compression and expansion unit 13. That is, the two line buffers A24 and B25 become alternately operative.
The image data read from the line buffers A24 and B25 are subjected to compression into a predetermined format such as JPEG at the image compression and expansion unit 13, and are thereafter sent as compressed image data 17 to the main memory 5. The compressed image data 17 is subjected to processing by an image display unit 19 and a signal conversion unit 20, whereby an image is displayed on a display device 21, or the compressed image data 17 is stored in a memory card through a memory card interface. The image display unit 19 uses a display buffer 29 as a working buffer memory.
Unlike the interlace sensor 2, the progressive sensor 18 as a CCD sensor for image capturing outputs the captured image of one frame at a time, and hence, generation of the original image data 16 on the main memory 5 is not required. Except for this, image processing using the progressive sensor 18 follows the same flow as the foregoing image processing using the interlace sensor 2.
As a result of higher pixel resolution, the CCD sensors 2 and 18 each have a growing number of pixels per horizontal line, reaching such a length that exceeds the horizontal line length of the line memory 8 of the RPU 7 in some cases. On the occurrence of such excess, regardless of whether the CCD sensor 2 or 18 is used, the image data outputted from the CCD sensor 2 or 18 cannot be directly processed by the RPU 7. In response, even in the use of the progressive sensor 18, the original image data 16 is generated once on the main memory 5, and thereafter the subsequent process steps are followed. More specifically, with reference to
As discussed, for compression of a subject image captured by a digital camera into a predetermined format such as JPEG to be displayed on a display device or stored in a memory card, image data transmission should be repeated between the main processor 4 and the main memory 5 through a bus 14. More specifically, a digital camera using the interlace sensor 2 requires five transmissions through the bus 14 as shown in
As discussed, the CCD sensors 2 and 18 have a horizontal line of pixels reaching a length that exceeds the horizontal line length of the line memory 8 of the RPU 7, in which case the regional data R1 and R2 on the left and right separately processed are synthesized into the intermediate data 26 which is then held in the main memory 5. As a result, the main memory 5 should be larger in capacity than required in the case in which the CCD sensors 2 and 18 contain pixels per horizontal line which do not reach the horizontal line length of the line memory 8 of the RPU 7.
The foregoing data transmissions are handled by a DMA control unit 15 which realizes high-speed DMA transmission through the bus 14 with no intervention of a CPU. However, when the CCD sensors 2 and 18 are a 5-megapixel-class CCD sensor with 2560 pixels high by 1920 pixels wide each storing 16 bits of information, for example, the image data outputted from the CCD sensor 2 or 18 contains information as much as about 10 megabytes. In light of a recent trend towards still higher pixel resolution of the CCD sensors 2 and 18, repetition of transmission of such high-volume data causes heavily places a burden on a bus band, thus leading to increase in power consumption, lower processing speed, and the like.
The present invention is intended for an image processing apparatus in which an image processing unit writes image data to a buffer memory, and a compression unit reads the image data from the buffer memory to perform compression thereon.
According to the present invention, the image processing apparatus comprises: an image processing part for performing image processing on captured data of an image; and a storage unit provided outside the image processing part. The image processing part includes: an image processing unit for performing a predetermined process on the captured data to obtain image data, and outputting the image data; a buffer memory for storing the image data sent from the image processing unit; and a compression unit for compressing the image data read from the buffer memory.
Accordingly, high-speed processing of the image data is realized without using the storage unit. Reduction in power consumption is allowed as well.
According to one aspect of the present invention, the buffer memory includes two buffer memories. The image processing apparatus further comprises a control unit for controlling read and writing to and from these two buffer memories. The control unit is operative in such a manner that while the image processing unit writes image data to either one of the two buffer memories, the compression unit selectively reads image data previously stored in the other one of the two buffer memories. As a result, the image processing unit and the compression unit are both allowed to perform sequential image processing without the need for waiting for completion of data transmission to the buffer memories.
It is therefore an object of the present invention to provide an image processing apparatus realizing high-speed image processing with low power consumption.
These and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.
The main processor 4 comprises an SPU 6, an RPU 7 including a line memory 8, an image compression and expansion unit 13, a local buffer A9 (first buffer memory) and a local buffer B10 (second buffer memory) serving to transmit data between the RPU 7 and the image compression and expansion unit 13, a write control unit 11 and a read control unit 12 for selecting which one of the two local buffers A9 or B10 to use, a bus 14 for data transmission, and a DMA control unit 15 for controlling data transmission through the bus 14 between the main processor 4 and the main memory 5.
The main memory 5 serves to temporarily store original image data 16 generated in the main processor 4 by data processing of odd and even fields sent from the interlace sensor 2 through the signal processing unit 3, and compressed image data 17 sent from the image compression and expansion unit 13.
Except for the two local buffers A9 and B10 provided between the RPU 7 and the image compression and expansion unit 13, and operation of the main memory 5 as a line buffer, the digital camera of the first preferred embodiment has the same configuration as that of the conventional digital camera.
Though not specifically shown, the main processor 4 further comprises a CPU for various types of operations, an external memory card for storing captured image data, hardware serving as an interface for storage into the external memory card, and a storage device for storing a software program.
The operation of the image processing apparatus 1 having the foregoing configuration will be discussed below.
In the main processor 4, the SPU 6 receives image data from the interlace sensor 2 through the signal processing unit 3 to generate the original image data 16 on the main memory 5. Next, the RPU 7 stores image data in the line memory 8 read from the original image data 16 and performs signal processing thereon such as interpolation, color space conversion and pseudo color suppression. These process steps are the same as those followed by the conventional digital camera discussed with reference to
More specifically, the processed image data of 8 lines is sent to the local buffer A9. The RPU 7 thereafter continues to process subsequent image data of 8 lines read from the original image data 16, and sends the processed image data to the local buffer B10. Concurrent with transmission of the image data from the RPU 7 to the local buffer B10, the image data stored in the local buffer A9 is sent to the image compression and expansion unit 13 for compression into a predetermined format such as JPEG. The compressed image data is thereafter sent to the main memory 5. The RPU 7 still continues to process subsequent image data of 8 lines read from the original image data 16 and send the processed image data to the empty line buffer A9. The image data stored in the local buffer B10 is concurrently sent to the image compression and expansion unit 13 for compression into a predetermined format such as JPEG. The compressed image data is thereafter sent to the main memory 5. That is, while the RPU 7 sends image data either to the local buffer A9 or to B10, image data stored either in the local buffer B10 or in A9 is sent to the image compression and expansion unit 13 to be compressed therein. These operations are repeatedly performed, whereby the compressed image data 17 is generated on the main memory 5.
The write and read control units 11 and 12 select which one of the two local buffers A9 or B10 to use by making connection to the RPU 7 and the image compression and expansion unit 13.
The line buffers A9 and B10 are only required to be not less than image data processed at one time by the RPU 7 in horizontal line length. More specifically, as long as the local buffers A9 and B10 have a horizontal line not less than one horizontal line of pixels of the CCD sensors 2 and 18, the local buffers A9 and B10 may be the same as or shorter than in horizontal line length the line memory 8 of the RPU 7. By way of example, when the CCD sensors 2 and 18 are a 5-megapixel-class CCD sensor with 2560 horizontal pixels, and the line memory 8 of the RPU 7 has a horizontal line length with 4096 pixels, the horizontal line length of the local buffers A9 and B10 may contain 2600 pixels or 4096 pixels, as long as it is not less than the length of 2560 pixels.
As discussed, the use of the local buffers A9 and B10 reduces the number of DMA transmissions of image data between the main processor 4 and the main memory 5 under control by the DMA control unit 15 until the compressed image data 17 is generated. That is, the conventional digital camera requires five transmissions in the use of the interlace sensor 2 and three transmissions in the use of the progressive sensor 18 as shown in
Under the conditions that the progressive sensor 18 is used as a CCD sensor, and the progressive sensor 18 has a horizontal line of pixels that does not exceed in length the horizontal line of the local buffers A9 and B10, the first preferred embodiment allows real-time data processing flow from the SPU 6 to the image compression and expansion unit 13 by way of the local buffers A9 and B10 without generating the original image data 16 once on the main memory 5 as shown in
In a second preferred embodiment of the present invention, it will be discussed what process a digital camera follows to handle the case in which the CCD sensors 2 and 18 have a horizontal line of pixels exceeding in length the horizontal line length of the local buffers A9 and B10.
As an example, assuming that the CCD sensors 2 and 18 are a 6-megapixel-class CCD sensor with 3072 pixels per horizontal line, that the line memory 8 of the RPU 7 contains 4096 pixels per horizontal line, and that the local buffers A9 and B10 each have 2600 pixels per horizontal line responsive only to a 5-megapixel-class CCD with 2560 pixels per horizontal line, the local buffers A9 and B10 are overflowed with data sent from the RPU 7 which prevents process flow. In response, in the second preferred embodiment, the two local buffers A9 and B10 together serve as one continuous buffer memory as shown in
The use of the local buffers A9 and B10 as one continuous buffer memory does not allow the RPU 7 and the image compression and expansion unit 13 to follow sequential process flow by means of the alternate use of the two local buffers A9 and B10 as discussed in the first preferred embodiment. In response, a control signal is transmitted between the RPU 7 and the image compression and expansion unit 13.
More specifically, the RPU 7 first processes image data of one line read from the original image data 16, and sends the processed image data to the buffer memory as a unity of the two local buffers A9 and B10. At the time when data transmission is completed, the RPU 7 stops the operation. The image compression and expansion unit 13 serves to process the data read from the buffer memory. After receipt of all the data in the buffer memory, the image compression and expansion unit 13 sends a control signal to the RPU 7 notifying completion of data readout. On receipt of this control signal, the RPU 7 again processes image data read from the original image data 16, and sends the processed image data to the buffer memory.
The second preferred embodiment is applicable even in the use of the progressive sensor 18 as a CCD sensor. More specifically, data from the SPU 6 is not directly sent to the RPU 7, but is sent to the main memory 5 once to generate the original image data 16.
In
The second preferred embodiment has been described in the use of the two local buffers A9 and B10 as discussed in the first preferred embodiment. As the simplest way to use a buffer memory, one local buffer may be provided in the image processing apparatus 1 that has the same capacity as that of the line memory 8 of the RPU 7. The second preferred embodiment is also feasible by the use of such a local buffer and the foregoing control signal.
As discussed, even when the CCD sensors 2 and 18 have high pixel resolution with a horizontal line of pixels exceeding in length the horizontal line length of the two local buffers A9 and B10, the foregoing use of one buffer memory as a unity of the local buffers A9 and B10 realizes image processing. Even when only one buffer memory is prepared which has a horizontal line length exceeding the length of one horizontal line of pixels of the CCD sensors 2 and 18, the use of the foregoing control signal realizes image processing. Like the first preferred embodiment, data transmission between the RPU 7 and the image compression and expansion unit 13 does not require the main memory 5 and the bus 14. A burden placed on a bus band is lightened accordingly, whereby high-speed image processing and reduction in power consumption are realized.
In the first and second preferred embodiments, the local buffers A9 and B10 serve to transmit data from the RPU 7 to the image compression and expansion unit 13. The local buffers A9 and B10 may be operative for another processing performed in the main processor 4. In a third preferred embodiment of the present invention, the local buffers A9 and B10 are operative for image display on a display device 21.
For visual recognition of a subject to be captured, a digital camera may use an electronic viewfinder or a liquid crystal display. With reference to
For the purpose of visual recognition of a subject to be captured, reproduction of the subject should be continuously made on the display device 21, and should not be limited to a capturing period. Accordingly, data should be continuously sent to the image display unit 19, thus heavily placing a burden on the bus 14 serving for this data transmission.
In response, in the third preferred embodiment, the local buffers A9 and B10 serve to transmit data to the image display unit 19 to lighten a burden on a bus band.
Image data sent from the CCD sensor 2 or 18 is subjected to the processing at the SPU 6. The processed image data is thereafter sent to the RPU 7 after generation of the original image data 16 in the use of the interlace sensor 2, whereas in the use of the progressive sensor 18, it is directly sent to the RPU 7. The captured image data from the RPU 7 is sent through the local buffers A9 and B10 to the image compression and expansion unit 13 and to the image display unit 19. The captured image data is subjected to the processing at the image compression and expansion unit 13 during image capturing, whereas in visual recognition of a subject to be captured, it is subjected to the processing at the image display unit 19. During visual recognition of a subject to be captured, display on the display device 21 is required to be at such a level that allows recognition of a subjected to be captured, and does not require data corresponding to millions of pixels sent from the CCD sensor 2 or 18. Accordingly, image data sent from the RPU 7 during visual recognition is skipped with 240 pixels high by 320 pixels wide, for example. The skipped image data is subjected to the sequential process flow as discussed in the first preferred embodiment including writing from the RPU 7 and reading to the image display unit 19 by means of alternate use of the two local buffers A9 and B10 under control by the write and read control units 11 and 12.
Continuity between the RPU 7 and the local buffers A9 and B10, and continuity between the local buffers A9 and B10, and the image compression and expansion unit 13 and the image display unit 19, are controlled by purpose-built hardware and a purpose-built software program having a switching function of continuity such as the write and read control units 11 and 12.
The output from the RPU 7 is sent to the local buffers A9 and B10, and the output from the local buffers A9 and B10 is selectively sent either to the image compression and expansion unit 13 or to the image display unit 19. That is, the local buffers A9 and B10 are selectively available either to the image compression and expansion unit 13 or to the image display unit 19. More specifically, between the image compression and expansion unit 13 and the image display unit 19, the local buffers A9 and B10 are allocated to the one placing a heavier burden on the bus 14.
As discussed, data from the RPU 7 is directly sent through the local buffers A9 and B10 to the image display unit 19 without data transmission through the bus 14 for image reproduction as required in the conventional digital camera. A burden placed on a bus band is lightened accordingly, to thereby reduce power consumption. The lightened burden on a bus band further advantageously leads to enhanced processing speed of the image processing apparatus 1 as a whole.
As discussed in the description of the background art with reference to
When the CCD sensors 2 and 18 have a horizontal line of pixels exceeding in length the horizontal line length of the line memory 8 of the RPU 7, and thus process flow cannot sequentially proceed from the SPU 6 to the RPU 7, the original image data 16 is generated once in the main memory 5 regardless of whether the CCD sensor 2 or 18 is used. Next, the original image data 16 is divided and is then subjected to the processing at the RPU 7 by means of the line memory 8. With reference to
As discussed, the RPU 7 processes the regional data including P1, P2, . . . , and the resultant data R1, R2, . . . are subjected to the processing at the image compression and expansion unit 13. Even when the CCD sensors 2 and 18 have a horizontal line of pixels exceeding in length the horizontal line length of the line memory 8, process flow continues without waiting for generation of intermediate data of one frame on the main memory 5 as required in the conventional digital camera. The fourth preferred embodiment further advantageously lowers working buffer capacity in the main memory 5. Still further, provision of two lines buffers A22 and B23 in the main memory 5, and alternate use of these buffers A22 and B23 as discussed with reference to the local buffers A9 and B10 in the first and second preferred embodiments realize image processing with still higher speed.
The CCD sensors 2 and 18 may have a horizontal line of pixels exceeding in length the line memory 8 of the RPU 7 as a result of higher pixel resolution of the CCD sensors 2 and 18, in which case the fourth preferred embodiment devises reading of image data from the original image data 16 to realize image processing. However, the fourth preferred embodiment requires a region for storing intermediate data in the main memory 5, thereby placing a heavier burden on the bus 14 and causing higher power consumption than the first and second preferred embodiments using the local buffers A9 and B10. A fifth preferred embodiment of the present invention as a combination of the second and fourth preferred embodiments is responsive to these problems.
As a circuit configuration of a digital camera, the fifth preferred embodiment employs the one shown in
More specifically, the original image data 16 generated on the main memory 5 after passing through the CCD sensor 2 or 18 and the SPU 6 is read as the regional data P1 and P2 in this order as shown in
The fifth preferred embodiment has been discussed as a combination of the fourth and second preferred embodiments. As long as the CCD sensors 2 and 18 have a horizontal line of pixels that does not exceed in length the horizontal line length of the local buffers A9 and B10, the first preferred embodiment may alternatively be combined with the fourth preferred embodiment. Still alternatively, a combination of the third and fourth preferred embodiments will be applicable.
As discussed, the use of the local buffers A9 and B10 of the second preferred embodiment combined with the fourth preferred embodiment eliminates the need for generating intermediate data on the main memory 5. Such combination further advantageously lightens a burden to be placed on the bus 14 and reduces power consumption.
While the invention has been shown and described in detail, the foregoing description is in all aspects illustrative and not restrictive. It is therefore understood that numerous modifications and variations can be devised without departing from the scope of the invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2003-093075 | Mar 2003 | JP | national |
This application is a Division of and claims the benefit of priority under 35 U.S.C. §120 from U.S. Ser. No. 10/811,840, filed Mar. 30, 2004, and claims the benefit of priority under 35 U.S.C. §119 from Japanese Patent Application No. 2003-093075, filed Mar. 31, 2003, the entire contents of each of which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5694166 | Fukushima | Dec 1997 | A |
| 6002446 | Eglit | Dec 1999 | A |
| 6137534 | Anderson | Oct 2000 | A |
| 6233010 | Roberts et al. | May 2001 | B1 |
| 6400471 | Kuo et al. | Jun 2002 | B1 |
| 6563535 | Anderson | May 2003 | B1 |
| 6847388 | Anderson | Jan 2005 | B2 |
| 6914632 | Kim | Jul 2005 | B1 |
| 20010019362 | Nakamura et al. | Sep 2001 | A1 |
| Number | Date | Country |
|---|---|---|
| 61-164377 | Jul 1986 | JP |
| 5-128232 | May 1993 | JP |
| 6-245013 | Sep 1994 | JP |
| 9-245157 | Sep 1997 | JP |
| 2002-125105 | Apr 2002 | JP |
| 2002-237951 | Aug 2002 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20120002080 A1 | Jan 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10811840 | Mar 2004 | US |
| Child | 13230068 | US |
