This U.S. non-provisional patent application claims priority under 35 U.S.C. §119 of Korean Patent Application No. 10-2012-0078317, filed on Jul. 18, 2012, the entire contents of which are hereby incorporated by reference.
1. Technical Field
The present inventive concept herein relates to a method of detecting a target in an image and an image processing device.
2. Discussion of the Related Art
Consumer demand for mobile imaging equipment such as a smart phone, a smart pad, a notebook computer, etc. increases rapidly. Various devices for generating and displaying multimedia contents are being introduced to the portable information equipment.
A camera is device typically used to generate multimedia content. A camera implemented as a digital still camera or a digital (video) camcorder obtains images and also may function as multipurpose equipment such as a smart phone and a tablet or smart pad.
In a portable information equipment such as a smart-phone/camera, various functions for increasing convenience of users are being developed. One of the convenient functions is target detection function for detecting a target in an image obtained using the camera. The target detecting function may be used to detect a person in the scene captured in the image and may be used as basis information for analyzing feelings and pose of a person in an image. The target detecting function detects the location of the target in an image and can be used as information for controlling a shooting direction, rate, and/or the optical focus distance of camera.
Exemplary embodiments of the inventive concept provide a method of detecting a target in an image. The method includes receiving an image; generating a plurality of scaled images on the basis of the received image; generating integral column images of the plurality of scaled images by calculating integral values of pixels column by column; classifying windows in the plurality of scaled images using the integral column images according to a feature arithmetic operation based on a recursive column calculation; and detecting the target on the basis of the windows classification results.
Exemplary embodiments of the inventive concept also provide an image processing device. The image processing device includes an image pyramid generating unit receiving an image and generating an image pyramid on the basis of the received image; a downscaling unit receiving the image pyramid and down-scaling each image of the image pyramid to output a plurality of images including the image pyramid and the downscaled image; a prefiltering unit outputting part of the plurality of the images on the basis of color maps of the plurality of images; an integral column generating unit receiving the part of the plurality of the images and performing an integral of each of the part of the plurality of the images by column unit to generate integral column images; a plurality of recursive column classifying units receiving the integral column images and classifying windows in the integral column images with respect to the received integral column image according to a feature arithmetic operation based on a recursive column calculation; and a clustering and tracking unit detecting a target in the image according to the windows classification result.
Embodiments of inventive concepts will be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. In the drawings, the size and relative sizes of layers and regions may be exaggerated for clarity. The embodiments of the inventive concept may, however, be embodied in different forms and should not be constructed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout.
Preferred embodiments of the inventive concept will be described below in more detail with reference to the accompanying drawings, in which:
Referring to
In step S130, integral column images of the plurality of images are generated respectively by calculating the added integral of pixels by column unit.
In step S140, integral column images are classified according to a feature arithmetic operation based on a recursive column arithmetic operation.
In step S150, the classified integral column images clusters to detect a target.
The preprocessing block 110 includes an image pyramid generating unit 111. The image pyramid generating unit 111 receives an image from the outside and generates an image pyramid on the basis of the received image. The image pyramid generating unit 111 generates images sequentially downscaled according to a previously set ratio and a previously set number. The image pyramid generating unit 111 generates images sequentially downscaled according to the ratio and the number set by a user.
The image pyramid generating unit 111 generates a first image having a size 1/n times as large as the received image size and a second image having a size 1/n times as large as the first image size. The image pyramid generating unit 111 generates the previously set number of downscaled images. The image pyramid generating unit 111 can output images including an original image and downscaled images.
The image pyramid generating unit 111 generates downscaled images but it is not limited thereto. The image pyramid generating unit 111 can generate an up-scaled image or an image to be up-scaled, and a downscaled image.
The image pyramid generating unit 111 generates all images according to the previously set ratio but it is not limited thereto. The image pyramid generating unit 111 can generate an image pyramid according to two or more ratios.
The image pyramid generating unit 111 can further generate a color map.
The image pyramid generating unit 111 can generate a color map of the original image or color maps of the original image and the downscaled images to output those color maps.
The main processing block 120 includes a caching unit 121, a downscaling unit 123, a prefiltering unit 125, an integral column generating unit 127, a feature caching unit 128, a control unit 129 and a plurality of recursive column classifying units 131 through 13k.
The caching unit 121 receives an image pyramid being output from the image pyramid generating unit 111 and stores the image pyramid. The caching unit 121 stores each image of the image pyramid by strip unit and can output the stored image by column unit.
The downscaling unit 123 receives an image from the caching unit 121 by column unit and generates intermediate images by column unit. The downscaling unit 123 generates images having a intermediate size of images generated by the image pyramid generating unit 111. The function of the downscaling unit 123 cooperates with that of the image pyramid generating unit 111.
The image pyramid generating unit 111, the caching unit 121 and the downscaling unit 123 perform the step S120 of
When the image pyramid generating unit 111 generate s color maps of the original image and the downscaled images, the downscaling unit 123 can scale the color maps to generate scaled color maps.
The prefiltering unit 125 can receive a plurality of scaled maps and scaled color maps from the downscaling unit 123. On the basis of the color maps, the prefiltering unit 125 can reject parts of the plurality of scaled images. The prefiltering unit 125 can reject parts of the scaled images on the basis of color and color change of the color maps. When a target to be detected is a person, the prefiltering unit 125 can reject images corresponding to color maps not having skin color. The prefiltering unit 125 outputs filtered images.
The integral column generating unit 127 receives filtered images from the prefiltering unit 125. The integral column generating unit 127 performs integral of pixel values of each image received by column unit to calculate integral values and can generate an integral column image having calculated integral values.
The feature caching unit 128 can store a plurality of features and transmits the stored features to the plurality of recursive column classifying units 131 through 13k. The feature caching unit 128 can transmit different features to the plurality of recursive column classifying units 131 through 13k.
The control unit 129 controls the overall operation of the main processing unit 120. The control unit 129 controls the prefiltering unit 125 so as to control a filtering object according to a detection target. The control unit 129 controls the feature caching unit 128 so that features are selected according to the detection target and the selected features are stored.
The control unit 129 controls the plurality of recursive column classifying units 131 through 13k so that a window is selected in the integral column image and a classifying operation is performed on the selected window. The control unit 129 controls the plurality of recursive column classifying units 131 through 13k so that features are selected on the basis of an adaptive boosting.
The plurality of recursive column classifying units 131 through 13k sequentially receive the integral column image from the integral column generating unit 127. Each of the plurality of recursive column classifying units 131 through 13k perform a feature arithmetic operation on the basis of the selected window of the integral column image. If the result of the feature arithmetic operation is FALSE, the corresponding window may be rejected. If the result of the feature arithmetic operation is TRUE, a next recursive column classifying unit performs a feature arithmetic operation on the basis of the selected window. The plurality of recursive column classifying units 131 through 13k can have different features.
The first recursive column classifying unit 131 performs a feature arithmetic operation on the basis of the selected window— If the result of the feature arithmetic operation is FALSE, the corresponding integral window may be rejected. If the result of the feature arithmetic operation is TRUE, a feature arithmetic operation can be performed on the basis of the window in which the second recursive column classifying unit 132 is selected.
The window that is determined as TRUE in all the plurality of recursive column classifying units 131 through 13k is transmitted to the post processor 150.
If a classification of the plurality of recursive column classifying units 131 through 13k with respect to the selected window is completed, a window is selected at a different location of the integral column image and a classification may be performed on the selected window.
If a classification is performed on all the integral column images, a classification of integral column image having a different size may be performed.
The plurality of recursive column classifying units 131 through 13k can operate in parallel. The plurality of recursive column classifying units 131 through 13k can receive an integral column image from the integral column generating unit 127 at the same time. The plurality of recursive column classifying units 131 through 13k can receive the same integral column image. The plurality of recursive column classifying units 131 through 13k can perform a feature arithmetic operation on the received integral column image. The feature arithmetic operations of the plurality of recursive column classifying units 131 through 13k can be performed at the same time.
If the feature arithmetic operations of the plurality of recursive column classifying units 131 through 13k are performed in cascade form, the size and complexity of the image processing device 100 may be reduced. One recursive column classifying unit is provided to the image processing device 100, different features are sequentially loaded on one recursive column classifying unit and a recursive feature arithmetic operation may be performed.
If the feature arithmetic operations of the plurality of recursive column classifying units 131 through 13k are performed at the same time, operation performance of the image processing device 100 may be improved. If the plurality of recursive column classifying units 131 through 13k performs a feature arithmetic operation at the same time, the speed that the image processing device 100 performs a feature arithmetic operation on a specific window increases.
The memory block 140 includes a memory 141. The memory 141 may include a random access memory (RAM). The memory 141 may include a volatile memory such as a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), etc. or a nonvolatile memory such as an electrically erasable and programmable ROM (EEPROM), a flash memory, a phase-change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The memory 141 may include a wideband I/O memory.
The memory 141 stores a plurality of features and can transmit the stored features to the feature caching unit 128. The memory 141 can transmit the selected features among the stored features to the feature caching unit 128 according to the control of the control unit 129.
The post processing block 150 includes a clustering and tracking unit 151. The clustering and tracking unit 151 collects windows determined as TRUE by the plurality of recursive column classifying units 131 through 13k and combines the windows to select or set the optimum window. The clustering and tracking unit 151 can track a target on the basis of the selected or set window. The clustering and tracking unit 151 can track a look, pose, feelings of the target on the basis of the tracked target.
The image processing device 100 can form a system-on-chip (SoC). Each constituent element of the image processing device 100 may be constituted by hardware of the system-on-chip (SoC), or may be formed by software being executed in the hardware or by a combination of the software and the hardware.
Each strip is divided into a plurality of parts according to its height. Each part divided according to its height is stored in a plurality of caches of the caching unit 121.
Referring to
Thus, a plurality of scaled images including the original image is generated by the image pyramid generating unit 111, the caching unit 121 and the downscaling unit 123.
The integral column generating unit 127 performs an integration of gray scale values of pixels to generate an integral column image.
The recursive column classifying unit performs a feature arithmetic operation. The selected window includes a selected area and an unselected area. The selected area may be a feature. The feature arithmetic operation may be the arithmetic operation of finding the sum of pixel values of the selected area. The recursive column classifying unit can more easily perform the feature arithmetic operation with the data of the integral column image than when using a conventional image.
In
The recursive column classifying unit performs a feature arithmetic operation by column unit of an integral column image. The recursive column classifying unit adds integral values of the lowermost row of each area of the feature in each column and can subtract integral values of the row just above the uppermost row of each area. The calculation result is the sum of pixel values corresponding to the feature.
The recursive column classifying unit, according to the feature type, collects the specific number of integral values of each column of an integral column image to perform a feature arithmetic operation. As illustrated in
In a column in which a feature does not exist, the recursive column classifying unit can collect and arithmetically operate the same integral value four times. Thus, in a column in which a feature does not exist, an arithmetic operation result is 0.
In a column in which a feature exists, the recursive column classifying unit can collect and calculate an integral value in a row just above each area of the feature and an integral value of the lowermost row of each area of the feature.
The recursive column classifying unit compares a calculation result with a reference range. If the calculation result is within the reference range, the classifying result of the selected window is determined as TRUE. If the calculation result is not within the reference range, a classifying result of the selected window may is determined as FALSE.
If the classifying result is determined as FALSE, the corresponding window may be rejected. Thus, that classification with respect to the corresponding window may be complete.
If the classifying result is determined as TRUE, a next step of classification with respect to the corresponding window is performed. Thus, another recursive column classifying unit may perform a classification using another feature.
The image processing device 100 selects features on the basis of an adaptive boosting.
If the classification of the second stage is determined as FALSE, the selected window is rejected. If the classification of the second stage is determined as TRUE, a classification of a next stage may be performed on the selected window.
Windows determined as TRUE in all the recursive column classifying units 131 through 13k are transmitted to the clustering and tracking unit 151.
If the selected window is rejected or if a classification of the selected window is completed, a window of different location may be selected in an integral column image. After that, a classification may be performed again on the new selected window. If a classification with respect to all the windows of the integral column image is completed, a classification may be performed in an integral column image having a different size.
As described above, if a classification is performed by column unit, various types of features may be used. By contrast, a classification using a conventional integral image has a limitation that it can use only a feature of a square shape. According to some embodiments of the inventive concept, an integral column image is generated and a feature arithmetic operation is performed by column unit of the integral column image. In any type of feature, one column has one-dimensional form having a beginning pixel and an end pixel. Thus, in any type of feature, in one column, a feature arithmetic operation may be performed simply through addition and subtraction. If the integral column image and the classification of column unit are performed, various types of features may be used and reliability of the image processing device 100 and the method of detecting a target is improved.
The caching unit 121, the downscaling unit 123, the integral column generating unit 127 and the recursive column classifying units 131 through 13k perform generation and classification of the cache, and perform the downscale and the integral column image by column unit. Thus, the image processing device 100 and the method of detecting a target in accordance with some exemplary embodiments of the inventive concept satisfy on-the-fly and provide an improved target detection speed.
In step S220, a selected window (e.g., the first window) is selected from the selected integral column image.
In step S230, a selected feature (e.g., the first feature) is selected.
In step S240, on the basis of the selected window and the selected feature, integral values are calculated by column unit (i.e., column by column).
In decision step S250, it is determined whether a calculation result is TRUE (i.e., within a range corresponding to the selected feature) or FALSE (i.e., outside of the range). If the calculation result is within a reference range corresponding to the selected feature, it is determined as TRUE and if the calculation result is not within the reference range, it is determined as FALSE. If the calculation result is FALSE, the selected window is rejected in step S255 and decision step S270 is performed. If the calculation result is TRUE, step decision S260 is performed.
In the decision step S270, it is determined whether the selected window is the last window. If the selected window is not the last window, a next window is selected (e.g., an index, not shown, is incremented) in step S275 and the step S230 is performed again with the new selected window. If the selected window is the last window, decision step S280 is performed.
In the decision step S280, it is determined whether the currently selected integral column image is the last integral column image. If the currently selected integral column image is not the last integral column image, in step S285, a next integral column image is selected (e.g., an index, not shown, is incremented) and the step S220 is performed again. If the selected integral column image is the last integral column image, a classification is completed.
When comparing the image processing device 200 with the image processing device 100 of
The power-OFF domain block 1100 is a block that is power-downed to realize a low power of the system-on-chip 1000. The power-ON domain block 1300 is a block that becomes power-ON to operate a part of functions of the power-OFF domain block 1100 while the power-OFF domain block 1100 is in a power-down state.
The power-OFF domain block 1100 includes a main central processing unit (CPU) 1110, an interrupt controller 1130, a memory controller 1120, first through nth intellectual properties (IP) 1142 through 114n and a system bus 1150.
The main central processing unit 1110 controls the memory controller 1120 to access the external memory 2000. The memory controller 1120 transmits data stored in the external memory 2000 to the system bus 1150 in response to a control of the main central processing unit 1110.
When an interruption (i.e., a specific event) occurs in each of the first through nth intellectual properties (IP) 1142 through 114n, the interrupt controller 1130 informs the main central processing unit 1110. The first through nth intellectual properties (IP) 1142 through 114n perform specific operations according to a function of the system-on-chip 1000. The first through nth intellectual properties (IP) 1141 through 114n access internal memories IP[#]_MEM 1361 through 136n respectively. The power-ON domain block 1300 includes the internal memories IP[#]_MEM 1361 through 136n of the first through nth intellectual properties (IP) 1142 through 114n.
The power-ON domain block 1300 includes a low power management module 1310, a wake-up IP 1320, a keep alive IP 1350 and the internal memories 1361 through 136n of the first through nth intellectual properties (IP) 1142 through 114n.
The low power management module 1310 determines whether to wake-up the power-OFF domain block 1100 according to data transmitted from the wake-up IP 1320. Power of the power-OFF domain block 1100 that is turned OFF during a standby state waiting an external input may be turned OFF. The wake-up is an operation applying a power supply again when data from the outside is input to the system-on-chip 1000 that becomes power-OFF. That is, the wake-up is an operation of making the system-on-chip in a standby state become an operation state (power-on state) again.
The wake-up IP 1320 includes a PHY 1330 and a LINK 1340. The wake-up IP performs an interface role between the low power management module 1310 and the external chip 3000. The PHY 1330 actually exchanges data with the external chip 3000 and the LINK 1340 transmits and receives the data actually exchanged in the PHY 1330 to and from the low power management module 1310.
The keep alive IP 1350 determines a wake-up operation of the wake-up IP 1320 to activate or deactivate electric power of the power-OFF domain block 1100.
The low power management module 1310 receives data from at least one IP of the first through nth intellectual properties (IP) 1142 through 114n. In the case that data is only transmitted without being processed, the low power management module 1310 stores the received data in an internal memory of the corresponding IP in place of the main central processing unit 1110.
The internal memories 1361 through 136n of the first through nth intellectual properties (IP) 1141 through 114n are accessed by corresponding IPs in the power-ON mode and are accessed by the low power management module 1310 in the power-OFF mode.
At one IP among the first through nth intellectual properties (IP) 1141 through 114n can correspond to the preprocessing block 110 or 210, the main processing block 120 or 220 and the post processing block 150 or 250 of the image processing device 100 or 200 in accordance with some exemplary embodiments of the inventive concept. At least one IP can include one of the preprocessing blocks 110 or 210, the main processing block 120 or 220 and the post processing block 150 or 250. The first through nth intellectual properties (IP) 1141 through 114n may include a graphic processing unit (GPU), a modem, a sound controller, a security module, etc.
At least one internal memory of the internal memories 1361 through 136n can correspond to the memory block 140 or 240 of the image processing device 100 or 200 in accordance with some exemplary embodiments of the inventive concept.
The image processing device 100 or 200 in accordance with some exemplary embodiments of the inventive concept can form the system-on-chip 1000.
The system-on-chip 1000 can form an application processor (AP0.
The application processor 4100 is configured to control an overall operation of the multimedia device 4000. The application processor 4100 can be formed of one system-on-chip (SoC). The application processor 4100 may include the system-on-chip 1000 described with reference to
The volatile memory 4200 may be an operational memory of the multimedia device 4000. The volatile memory 4200 may include a dynamic random access memory (DRAM) or a static random access memory (SRAM).
The nonvolatile memory device 4300 may be a main storage place of the multimedia device 4000. The nonvolatile memory 4300 may include a nonvolatile storage device such as a flash memory, a hard disk drive, a solid state drive, etc.
The one or more input/output controllers 4400 are configured to control the one or more input/output devices 4500.
The one or more input/output devices 4500 may include various devices receiving a signal from the outside. The one or more input/output devices 4500 may include a keyboard, a keypad, a button, a touch panel, a touch screen, a touch pad, a touch ball, a camera including an image sensor, a microphone, a gyroscope sensor, a vibration sensor, a data port for a wire input, an antenna for a wireless input, etc.
The one or more input/output devices 4500 may include various devices outputting a signal to the outside. The one or more input/output devices 4500 may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display device, an active matrix OLED (AMOLED) display device, an LED, a speaker, a motor, a data port for a wire output, an antenna for a wireless output, etc.
The multimedia device 4000 obtains an image that may include a target and can perform an integral arithmetic operation of column unit (column by column) on the obtained image. The multimedia device 4000 tracks a target using various features and can track a pose, feelings and an atmosphere of the target.
The multimedia device 4000 may be implemented as a mobile multimedia device such as a smart phone, a smart pad, a digital camera, a digital camcorder, a notebook computer, etc. or a fixed multimedia device such as a smart television, a desktop computer, etc.
According to an aspect of the inventive concept, a feature arithmetic operation is performed by column unit using an integral column image. Thus, various types of features may be detected and a method of detecting a target and an image processing device having improved target detecting function and speed are provided.
The foregoing is illustrative of the inventive concept and is not to be construed as limiting thereof. Although a few exemplary embodiments of the inventive concept have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings of the present inventive concept. Accordingly, all such modifications are intended to be included within the scope of the present invention as defined in the claims. The present invention is defined by the following claims, with equivalents of the claims to be included therein.
Number | Date | Country | Kind |
---|---|---|---|
10-2012-0078317 | Jul 2012 | KR | national |