The disclosure relates to the field of image processing and more particularly to image filtering at the pixel level.
Image processing techniques are employed by many applications such as, but not limited to, gesture detection and recognition. In short-range applications, for example, fingers can be located to determine hand gestures performed by a user. Similarly, in long-range applications, images can be filtered to locate limbs to track or identify a user's body movements. In some applications, certain objects can be similarly tracked or identified by processing an image or a series of sequentially collected images. Improved image processing systems and techniques are needed to meet performance and precision requirements of modern applications.
An embodiment of the disclosure includes a system for image processing to locate portions of an image exhibiting high singularity (i.e. singular points) which are attributable to features such as, but not limited to, fingers, hands, feet, limbs, bodies, facial features, certain objects, or portions thereof. The system includes a storage module in communication with a plurality of memory banks. The storage module is configured to store image pixels in the memory banks and further configured to interleave the memory banks, thereby enabling a plurality of image scanners to access the image pixels in parallel. The system further includes a scanning module including the plurality of image scanners. The scanning module is configured to scan a selection of pixels in at least four directions relative to a first pixel utilizing the plurality of image scanners. The system further includes a singular points detection module configured to acquire a depth of each pixel of the selection of pixels scanned by the scanning module. The singular points detection module is further configured to determine a singularity value of the first pixel by comparing the depth of the first pixel with the depth of each pixel of the selection of pixels within a selected proximity of the first pixel.
It is to be understood that both the foregoing general description and the following detailed description are not necessarily restrictive of the disclosure. The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the disclosure.
The embodiments of the disclosure may be better understood by those skilled in the art by reference to the accompanying figures in which:
Reference will now be made in detail to the embodiments disclosed, which are illustrated in the accompanying drawings.
The system 100 includes a plurality of modules including hardware, software, firmware, or any combination of the foregoing configured to execute various functions or steps described herein. In some embodiments, one or more of the various functions or steps are carried out by at least one processor, such as a single-core or multiple-core processor, executing program instructions from carrier or storage media. In some embodiments, a module further includes dedicated hardware such as, but not limited to, a microcontroller, ASIC, FPGA, electronic circuitry, or combinational logic configured to execute one or more of the various steps or functions described herein.
According to various embodiments, the system 100 includes a storage module 102 in communication with a scanning module 105. In some embodiments, the scanning module 105 includes a plurality of scanning sub-modules 106 (hereinafter “image scanners 106”), each configured to independently scan image pixels along a selected direction. The storage module 102 is configured to store image pixels on at least one storage medium 104. The storage module 102 is further configured to interleave memory banks storing the image pixels to enable the scanning module 105 to scan a selection of image pixels in a plurality of directions with two or more image scanners 106 operating in parallel. In some embodiments, the storage module 102 is configured to enable up to four image scanners 106 operating in parallel to scan the selection of image pixels in at least four directions relative to the first image pixel. In some embodiments, the scan directions include at least a vertical direction, a horizontal direction, and two diagonal directions allowing the scanning module 105 to scan a selection of pixels proximate to the first pixel. As used herein, the “first pixel” refers to any pixel of interest and is not limited to one pixel of the image.
The system 100 further includes one or more image filtering modules 107 in communication with the scanning module 105. In some embodiments, the filtering modules 107 include a singular points detection module 108 configured to determine singularity values for one or more image pixels. In some embodiments, the singular points detection module 108 is configured to determine a singularity value of one or more image pixels such as the first pixel by comparing depth of the first pixel with depth of each pixel of the selection of scanned pixels in proximity of the first pixel. In some embodiments, the singular points detection module 108 is further configured to determine a singularity value for each pixel of a plurality of image pixels iteratively processed by the scanning module 105 in a manner similar to the first pixel.
In some embodiments, the image filtering modules 107 include one or more image filters 109 instead of or in addition to the singular points detection module 108. Many additional filters 109 are known to the art such as, but not limited to, Gaussian, median, and bilateral filters. In some embodiments, the one or more image filtering modules 107 are configured to determine at least one attribute (e.g. singular points) of the image by processing scan data collected by the one or more image scanners 106 from the selection of scanned pixels. In some embodiments, the one or more image filtering modules 107 are additionally or alternatively configured to generate a filtered version of the image based upon the scan data.
In an embodiment, illustrated in
In some embodiments, the arbiter 110 is configured to resolve memory access conflicts utilizing a scheduling technique such as, but not limited to, a “round robin” conflict resolution technique. For example, a first image scanner 106 and a second image scanner 106 of the scanning module 105 may simultaneously request access to (same or different) pixels stored in a first memory bank 112. In some embodiments, the arbiter 110 is configured to provide the first image scanner 106 access to the first memory bank 112 while the second image scanner 106 remains idle. At a succeeding clock cycle, the first image scanner 106 moves to a completion state or requests image pixels from a second memory bank 112. The arbiter 110 is then configured to provide the second image scanner 106 access to the first memory bank, thereby allowing the first and second image scanners 106 to scan an image corresponding to the image pixels in parallel without memory access conflicts. In some embodiments, two or more image scanners 106 operating in parallel are further configured to scan the image in different directions by accessing the interleaved memory banks 112 via sequential access ports of the arbiter 110.
In some embodiments, the storage module 102 further includes a synchronizer 114 configured to synchronize memory access for two or more image scanners 106 to support various filters such as, but not limited to, Gaussian, median, or bilateral filters. The synchronized access allows the scanning module 105 to collect information about the selection of pixels proximate to (i.e. surrounding) the first pixel at the same clock cycle for each image scanner 106. In some embodiments, the synchronizer 114 is further configured to provide an indication (e.g. “pixels_done” output) when all scanners processing the image are finished scanning a first selection of pixels. Upon receiving the indication from the synchronizer 114, the image scanners 106 are configured to begin processing a second selection of pixels unless the image or a selected portion of the image has been completely processed. In some embodiments, the image scanners 106 are configured to substantially simultaneously move on to the second selection of pixels.
For short-range applications, the depth (i.e. distance to camera) value for each pixel of a corresponding image can be represented as an 8-bit value. Increasing internal bus width up to a selected software interface bus width (e.g. 32-bit width) can improve overall performance of the system 100. In some embodiments, the storage module 102 is thus configured to store four 8-bit depth values in each of a plurality of 32-bit memory banks 112 accessed via 32-bit buses. In long-range applications, the depth of each pixel can be represented by a greater number of bits (e.g. 16-bit value). In some embodiments, the storage module 102 is thus configured to store two 16-bit depth values in each of the plurality of 32-bit memory banks 112. The system 100 may employ any selected number of bits to represent the pixel depth values with wider buses and memory cells to improve performance.
In some embodiments, the image storage module 102 includes the following (Verilog) interface script or functionally equivalent code:
The scanning module 105 is configured to scan image pixels on a pixel by pixel basis via access ports of the storage module 102. In some embodiments, the scan direction of one or more image scanners 106 of the scanning module 105 is configurable at runtime (e.g. by changing a “direction” input). Accordingly, pixels can be scanned in multiple directions utilizing the same image scanner 106 for different filters or for different stages of a filter. In some embodiments, the scanning module 105 further includes a first-in-first-out (FIFO) submodule configured to store a selected number of previously scanned pixels while additional pixels are being scanned. The FIFO submodule reduces the number of memory access conflicts when implementing filters with a large carrier because the previously scanned pixels are temporarily stored for access by a filter module, such as the singular points detection module 108. Accordingly, a number of scan directions required for processing each new pixel of interest is reduced (i.e. less scanner access to memory is required).
In some embodiments, the scanning module 105 and the storage module 102 include interfaces enabling the modules to be operatively chained in a pipeline such that a first stored image is scanned for processing by a first filter, a second stored image is scanned for processing by a second filter, and so on. In some embodiments, the scanning module 105 includes the following (Verilog) interface script or functionally equivalent code:
In some embodiments, the singular points detection module 108 is configured to detect portions of the image exhibiting selected levels of singularity (i.e. singular points) that are attributable to features such as fingers or certain objects, among others. In some embodiments, the singular points detection module 108 is configured to detect a singular point at a first pixel having coordinates (X0, Y0) when for each pixel of a selection of 8 or more pixels surrounding the first pixel (in eight directions relative to the first pixel), there exists another pixel having coordinates (Xi, Yi) for which, according to Cartesian metrics:
(X0−Xi)2+(Y0−Yi)2≦R2, and
Di−D0≧H; or
according to Manhattan metrics:
|X0−Xi|+|Y0−Yi|≦R], and
Di−D0≧H;
where R is a selected radius (in pixels), Di is a pixel depth (e.g. distance from the ith point to a camera), and invalid pixels are assumed to have D=∞. Parameters R and H are selected according to the application. In short-range (e.g. ≦0.6 m) detection of fingers, for example, H=10 mm and R=5 are acceptable parameter values for normalized and original size images. In other embodiments, however, it is appreciated that alternative R and H values are likely required to conform to specification of a selected short-range or long-range application.
In some embodiments, the singular points detection module 108 is configured to determine a singularity value for each image pixel by comparing depth of each pixel with a selection of scanned pixels extending in a plurality of directions, as illustrated in
In some embodiments, up to four image scanners 106 operating in parallel are enabled to access image pixels in memory without conflicts, further improving processing speed. In some embodiments, a filter 108 or 109 is configured to access scan data according to a selected number of clock cycles such as, but not limited to, once every clock cycle or once every two cycles. In some embodiments, when scan data is accessed once every two or more clock cycles more than four image scanners 106 operating in parallel are enabled to access four memory banks 112 without conflicts. For example,
In some embodiments, the singular points detection module 108 is configured to execute the following steps at each clock cycle or at a selected number of clock cycles. The detection module 108 is configured to acquire a depth Din of a next (incoming) pixel in a current direction and a depth Dout of a previous (outgoing) pixel stored by the FIFO submodule. The detection module 108 is further configured to determine a minimum depth Dmin and a maximum depth Dmax of pixels stored by the FIFO submodule. The detection module 108 is further configured to: increase the singularity value of each pixel in the FIFO submodule for which Din−Di≧H, when Din−Dmin≧H; increase the singularity value of each pixel in the FIFO submodule for which Dout−Di≧H, when Dout−Dmin≧H; increase the singularity value of the incoming pixel, when Dmax−Din≧H; and increase the singularity value of the outgoing pixel (from FIFO), when Dmax−Dout≧H.
In some embodiments, the foregoing approach enables the steps illustrated in
The singular points detection module 108 is further configured to detect one or more singular points of the image based upon singularity values determined for the scanned pixels. As illustrated in
In some embodiments, illustrated in
The gesture recognition module 118 is configured to detect various gestures based upon singular points detected in one or more images by the singular points detection module. In some embodiments, the gesture recognition module 118 is configured to detect a gesture based upon the location of at least one singular point in one image or based upon a plurality of locations tracked through a series of images. In some embodiments, the gesture recognition module 118 is further configured to associate one or more singular points with one or more features (e.g. fingers) and further configured to detect a gesture based upon the location of the one or more features within the image or a change in location tracked through a series of images. In some embodiments, the gesture recognition module 118 is further configured to associate a first set of one or more singular points with a first feature (e.g. finger) based upon a first singularity threshold and a second set of one or more singular points with a second feature (e.g. hand) based upon a second singularity threshold. The gesture recognition module 118 may accordingly detect gestures based upon a plurality of features located throughout an image or tracked through a series of images.
At step 202, pixels corresponding to an image are stored in a plurality of memory banks. At step 204, the memory banks are interleaved to enable a plurality of image scanners operating in parallel to read the image pixels. At step 206, the image is scanned in at least four directions relative to a first pixel utilizing two or more image scanners operating in parallel. At steps 208 and 210, a depth value of the first pixel is compared against depth values corresponding to a selection of pixels proximate to the first pixel (e.g. all pixels within radius R). A singularity value is assigned to the first pixel based upon the comparison. The foregoing steps are further repeated as the image is scanned until singularity values are assigned to all pixels within the image or a selected portion of the image. In some embodiments, singular points attributable to various features are detected according to one or more threshold singularity values. In some embodiments, at least one gesture is detected utilizing information associated with the singular points such as the location of one or more singular points in an image or through a series of images.
It should be recognized that in some embodiments the various functions or steps described throughout the present disclosure may be carried out by any combination of hardware, software, or firmware. In some embodiments, various steps or functions are carried out by one or more of the following: electronic circuits, logic gates, field programmable gate arrays, multiplexers, or computing systems. A computing system may include, but is not limited to, a personal computing system, mainframe computing system, workstation, image computer, parallel processor, or any other device known in the art. In general, the term “computing system” is broadly defined to encompass any device having one or more processors, which execute instructions from a memory medium.
Program instructions implementing methods, such as those manifested by embodiments described herein, may be transmitted over or stored on carrier medium. The carrier medium may be a transmission medium, such as, but not limited to, a wire, cable, or wireless transmission link. The carrier medium may also include a storage medium such as, but not limited to, a read-only memory, a random access memory, a magnetic or optical disk, or a magnetic tape.
It is further contemplated that any embodiment of the disclosure manifested above as a system or method may include at least a portion of any other embodiment described herein. Those having skill in the art will appreciate that there are various embodiments by which systems and methods described herein can be effected, and that the implementation will vary with the context in which an embodiment of the disclosure deployed.
Furthermore, it is to be understood that the invention is defined by the appended claims. Although embodiments of this invention have been illustrated, it is apparent that various modifications may be made by those skilled in the art without departing from the scope and spirit of the disclosure.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/RU2013/000321 | 4/16/2013 | WO | 00 |