Patent Application
20200372234
This invention relates generally to quality determination and segmentation. More specifically, an exemplary aspect of the invention relates to quality and segmentation determination of biometric image information, such as fingerprint, handprint or footprint image information.
Livescan type devices provide preview modes of operation that serve as an aid to the operator for finger placement and image fidelity. Because of processing power, timing restrictions and bandwidth requirements, these devices and their associated quality metrics are unable to provide output that can be correlated to other more sophisticated measures of quality used by enrollment applications or Automatic Fingerprint Identification Systems (AFIS), and their associated matching engines. The limitations of the hardware also contributes to deficiencies in the ability to dynamically segment the fingertips from the rest of the hand structure and to separate the image background, such as, smudges, particulates, latent prints, and the like from the fingertips.
As a substitute, livescan type devices utilize image contrast based quality metrics and static scanner platen areas for segmentation. Fingerprint quality metrics used within enrollment and AFIS/matching applications are typically not suitable for this preview capability due to their requirements for full resolution scanning of the fingerprint imagery, typically at 500 ppi and 1000 ppi.
One exemplary aspect of the invention addresses this gap by providing a resolution independent ridge flow detection technique. A ridge flow map is then used to determine fingerprint quality via a combination of ridge flow strength, ridge flow continuity, and localized image contrast. This ridge flow map can also be used as a basis for segmentation.
Another exemplary aspect of the invention is a ridge flow based system and methodology for fingerprint image quality determination that is independent of image resolution, can be processed in real-time, and includes segmentation, such as fingertip segmentation, therefore providing image quality assessment for individual fingertips with a four finger flat, dual thumb, or whole hand image.
For example, one exemplary aspect is targeted at enrollment applications using livescan fingerprint devices that provide low-resolution preview image data, such as image data representing a finger, hand or other biometric information that includes ridge flow.
Aspects of the invention additionally relate to a fingerprint quality module that receives from one or more scan devices ridge-containing imagery which can then be assessed for one or more of quality, handedness, historical information analysis and the assignment of bounding boxes.
Additional exemplary aspects of the invention relate to ridge flow determination.
Further exemplary aspects of the invention relate to segmentation of a fingerprint image.
Additional exemplary aspects of the invention relate to determining a quality of an input image, such as a fingerprint image.
Still further aspects of the invention relate to modifying the ridge flow process based on image resolution.
Still further exemplary aspects of the invention relate to determining and providing feedback based on a received image(s).
Still further aspects of the invention relate to identifying one or more fingertips and, if needed, trimming the tips at an inter-digit crease.
Still further aspects of the invention relate to determining and placing a bounding box around an identified fingertip.
Additional exemplary aspects of the invention relate to determining in real-time fingerprint quality at a live scan type device.
These and other features and advantages of this invention are described in, or will become apparent from, the following detailed description of the embodiments.
The embodiments of the invention will be described in detail, with reference to the following figures, wherein:
The exemplary systems and methods of this invention will be described in relation to image quality and segmentation. However, to avoid unnecessarily obscuring the present invention, the following description omits well-known structures and devices that may be shown in block diagram form, are generally known or otherwise summarized. For purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It should however be appreciated that the present invention may be practiced in a variety of ways beyond the specific detail set forth herein.
Furthermore, while the exemplary embodiments illustrated herein show the various components of the system collocated, it should be appreciated that the various components of the system can be located at distant portions of a distributed network, such as a LAN and/or the internet, or within a dedicated system. Thus, it should be appreciated that the components of this system can be combined into one or more devices or collocated on a particular node of a distributed network, such as a communications network. It will be appreciated from the following description, and for reasons of computational efficiency, that the components of the system can be arranged at any location within a distributed network without affecting the operation of the system.
Furthermore, it should be appreciated that the various links connecting the elements can be wired or wireless links, or any combination thereof, or any other known or later developed element(s) that is capable of supplying and/or communicating data to and from the connected elements. These wired or wireless links can also be secure links and may be capable of communicating encrypted information.
The term module as used herein can refer to any known or later developed hardware, software, or combination of hardware and software that is capable of performing the functionality associated with that element. Also, while the invention is described in terms of exemplary embodiments, it should be appreciated that individual aspects of the invention can be separately claimed. While the embodiment discussed herein will be directed toward fingerprint quality and segmentation, it should also be appreciated that the systems and methods will work equally well for any type of image that includes any type of periodic or repeating pattern such as ridge flow including, but not limited to images of fingers, hands, feet or any portion thereof.
The fingerprint quality module 100 receives input from various sources. For example, the flatbed scanner or automatic document scanner 15 digitizes information on a fingerprint card 10 with a corresponding image being forwarded to the fingerprint quality module 100 for review. The live-scan type devices such as the whole-hand scanner 20, which can take a slap scan, the individual fingerprint scanner 25, and slap scanner 30 also capture image information corresponding to a scan of a ridge flow containing portion of the anatomy. Similarly, the single fingerprint roll scanner 35 inputs image data corresponding to a roll print to the fingerprint quality module 100.
Additionally, or alternatively, the fingerprint quality module 100 can receive image data from other optical or non-optical based scanners 40 such as capacitive, ultrasound, or in general from any source capable of forwarding an image containing ridge flow information.
In general, any of the input image(s) can be optionally pre-processed to enhance ridge flow detection. For example, the input image(s) can be pre-processed to adjust contrast, sharpness or sampling. The image(s) can also be preprocessed for enhancement or skeletonization.
In operation, the fingerprint quality module 100 receives an image from one of the input devices and determines ridge flow. Once ridge flow has been determined, if it can be determined, regions within the image can be identified, such as a finger, a finger tip, a toe, a hand, palm, or in general any region or regions based, for example, on the particular operating environment of the system 1.
Next, the fingerprint quality module 100 performs one or more of determining and scoring the quality of the image, assigning bounding boxes to the identified regions, determining handedness of the image and performing historical information analysis on a series of captured images. For example, images can be received in real-time at the fingerprint quality module 100. A series of images can be stored with the cooperation of storage 60 such that the fingerprint quality module 100 can perform an analysis over several frames of a scan and thus, for example, establish trending information regarding the quality of the images. This trending information can then be used to assist with providing audio or video feedback to a fingerprintee and/or dynamically utilized with predication of when a “good image” could be captured.
Depending on the results of one or more of the quality determination, assigning the bounding boxes, handedness determination and historical information analysis, the fingerprint quality module 100 can provide feedback via one or more of the audio and/or video feedback device(s) 45 and user interface(s) 55. For example, the audio and/or video device(s) 45 can be used to provide feedback to the fingerprintee to indicate if a scan has been successful, if movement of the finger(s) on the platen is required, to press harder or lighter, to straighten the hand, or the like. The feedback can be in audio and/or video and/or graphically, with audio cues and/or in a multitude of languages as needed based on, for example, the particular operating environment of the system 1.
Furthermore, the fingerprint quality module 100 could provide feedback via user interface 55. For example, user interface 55 can be used by the system operator with the feedback information including, for example, instructions indicating that the platen needs cleaning, there is a problem with one or more of the scanning devices, whether or not a good scan and capture of the fingerprint information was successful, operating status about the system, or in general any information related to the operation or information being processed by the fingerprint quality module 100.
The user interface 55, in cooperation with the demographic data source 65 also allows one or more of demographic data, other biometric data, or in general any information to be associated with one or more images received from the various scan devices. This information can include, but is not limited to, name, passport number, social security number, date and time information associated with the scan, location information of the scan, or in general any information that could, for example, be used to assist in identifying, classifying, categorizing or flagging the image. The image, with the demographic metadata, can then be stored, for example, in storage 60 or output via output 50.
Output 50 allows, for example, the fingerprint quality module to output one or more images with or without the demographic metadata to allow, for example, printing, off-line processing, identification, analysis, comparison, archiving, or in general output for any need.
As discussed hereinafter in greater detail, an exemplary non-limiting methodology is presented that performs segmentation of fingertips from a multi-finger impression image.
As previously discussed, the process starts with the receipt of a fingerprint image. Upon receipt of the image, and with the cooperation of the ridge flow module 190, the strength, continuity, and angle of ridge flow at, for example, each pixel of the fingerprint image is determined. Alternatively, or in addition, the strength, angle and continuity can be determined for predetermined positions within the image such as across a predetermined grid. While a specific methodology for ridge flow will be discussed hereinafter, it should be appreciated that any methodology can be used to determine ridge flow that is capable of outputting information that represents one or more of the strengths, continuity and angle of ridge flow at various points within the received image.
The segmentation module 120 performs the necessary determinations to identify fingertip locations. There are various methodologies that can be used for segmentation with, in general, the only requirement being accurate identification of the bounding box for finger portions.
The quality determination module 180 utilizes the ridge flow information and segmentation information to determine a quality metric based on a parsing of the fingerprint image area for one or more of contrast, ridge flow strength and local ridge flow. These factors can be combined to form a quality score for each tip. Based on this quality score, and in cooperation with the feedback determination module 165, appropriate feedback, if needed, can be determined and forwarded to an appropriate destination.
The sampling module 130 allows up or down-sampling of received images. For example, depending on the resolution of the received image, it may be advantageous from a computational standpoint to either up or down-sample the image.
The sequence checking module 170 performs a sequence check of the received fingerprint images. A typical sequence check would verify that the images corresponding to individual fingerprint scans are received in a particular order such as, for example, from the index finger to the pinky. If the pinky finger is determined to have been received out of sequence, or if a finger was errantly imaged twice, and in cooperation with the feedback module 165, appropriate feedback can be determined and forwarded to the appropriate destination.
The auto-capture module 197 works in conjunction with the quality determination module 180 and is capable of dynamically determining when an image is expected to meet a certain quality metric. For example, and in conjunction with the historical information analysis, trending information can be analyzed for prediction of when the fingerprint quality system, and in particular the fingerprint quality module 100 is expected to receive a “good” quality scan from one of the scanning devices. For example, if a fingerprintee is initialing pressing too lightly on the platen, and the trending information indicates that there has been a continuous increase in pressure on the platen, there will be an ideal point at which the fingerprint should be captured. This auto-capture module 197 thus tracks this historical information and captures images at a particular point in time during the image capture sequence.
In step S320, one or more regions of the received image are identified. These regions can include, but are not limited to, a finger, a fingertip, toe, hand, palm, or in general any region for which the fingerprint quality system has been assigned to identify. Control then continues to one or more of steps S325, S335, S340 and S345.
More specifically, in step S325, image quality is determined. Once the quality of the image has been determined, in step S330, the quality can be scored, for example, for each fingertip. Control then continues to step S350.
In step S335, one or more bounding boxes can be assigned to the identified region(s) with the bounding boxes being placed around the identified regions. More specifically, bounding boxes identify a region of interest about a finger, hand, palm, foot feature, etc., that is of particular interest to the system. Bounding boxes can be determined through use of one or more of ridge flow data, user input, or other data deemed applicable by the system. The exact extents of the box can be the minimum region of the feature of interest or of any further extend as required by the system.
In step S340, a handedness determination is performed. This allows the system to determine whether, for example, a left or right hand, or portion thereof, are represented in the received image. Control then continues to step S350.
In step S345, historical information analysis can be performed. As previously discussed, this historical information analysis allows the system to dynamically track trending information related to a plurality of received images and, for example, trigger a particular action based on one or more trends. Control then continues to step S350.
In step S350, a determination is made, based on any of the preceding steps, whether feedback should be provided to one or more of a system user and a fingerprintee. If a determination is made that feedback is needed, control continues to step S355 where that feedback is provided. Otherwise, control jumps to step S360 where a determination is made whether additional scanning is required. If additional scanning is required, control loops back to step S305. Otherwise, control continues to step S365 where the image(s) is stored. Control then continues to step S370 where the control sequence ends.
The exemplary methodology presented above can be used in a resolution independent manner. Thus, the various methodologies discussed herein can be modified to accept image data at various resolutions with the following detailed discussions of specific ridge flow, segmentation and quality calculations being capable of working with various versions of the same fingerprint image where the difference between the versions is the capture resolution. The processing of each version thus results in similar, if not the same, ridge flow detections. To assist with maintaining the fidelity of the process, both the selected segmentation processes and quality determinations should adhere to resolution invariance. The segmentation and quality methodologies described herein will be presented in a resolution invariant manner to support this fidelity.
It is the resolution independence and the high degree of correlation between the sub-resolution output and the full resolution output, in all aspects of processing-ridge flow segmentation and quality score, as well as the ability to process this information in real-time, that allows rapid feedback to be determined. By determining in a rapid fashion the ridge flow based quality and fingertip locations in a finger, the fingerprint quality system is able to capitalize on image capture devices that often generate low-resolution preview data.
In step S440, the peak ridge flow angle (if any) in the sub-windows is determined from the correlation map. Next, in step S450, optional smoothing or averaging of the raw ridge flow map can be performed. Control then continues to step S460 where the control sequence ends.
In step S520, the orientation for the larger cluster(s) is determined. Next, in step S525, the dominant orientation angle is determined. Then, in step S530, clusters are grouped that can be connected via a suspected segment orientation angle into segment cluster group(s). Control then continues to step S535.
In step S535, any non-segment cluster groups are removed. Next, in step S540, the end cluster of each segment cluster group is identified as the tip. Next, in step S545, a determination is made whether the identified tip is too long. If the identified tip is too long, control continues to step S550 where the tip is trimmed with control continuing to step S555. Otherwise, if the identified tip is not too long, control jumps to step S555 where a bounding box is determined for each tip. Control then continues to step S560 where the control sequence ends.
In step S620, the orientation for the larger cluster(s) is determined. Next, in step S625, the dominant orientation angle is determined. This dominant orientation angle can correspond to, for example, the hand. The orientation for the larger clusters generally corresponds to the clusters representing portions of a finger, such as a knuckle. In step S630, clusters are grouped that can be connected via the finger orientation angle. Next, in step S635, any non-finger cluster groups are removed. Then, the top-most cluster of each finger is identified as the fingertip. Control then continues to step S645.
In step S645, a determination is made whether the identified fingertip is too long. If the tip is too long, control continues to step S650 where the tip is trimmed at the inter-digit crease. Control then continues to step S655.
If the identified tip is not too long, control jumps to step S655 where the bounding box for each tip is determined. Control then continues to step S660 where the control sequence ends.
In step S715, the average contrast for the identified detections is determined. Next, in step S720, the peak Fourier coefficients for each ridge flow detection are determined. Then, in step S725, the peak angle agreement between each ridge flow detection and neighbors is determined. Control then continues to step S730.
In step S730, the results are combined and scaled into a quality score. Next, in step S735, a determination is made as to whether all fingertips have been processed. If all fingertips have been processed, control continues to step S750 where the control sequence ends. Otherwise, control jumps to step S740.
In step S740, the next target fingerprint is selected with control returning back to step S705.
In a more specific embodiment, a first step in the determination of quality is identification of all ridge flow detections that belong to the target fingertip. This information comes directly from the segmentation process. For each fingertip, all identified blocks are processed for image contrast, strength of detection, and angle agreement. The image contrast can be derived from the DC component of the Fourier analysis. The difference between the value of the current block and the image background can be the basis for this metric. The greater the difference, the greater the contrast, and the higher the quality score.
The second metric is the relative strength of the peak detection. The value of the Fourier coefficient can be compared against an ideal value. The stronger the coefficient, the more pronounced the ridge flow and the higher the quality score.
The final determination is the local agreement of peak detected angles. For each block, the detected angles are compared to the neighboring peak angles. If the majority of the angles match, then the resulting quality score is high.
These three metrics for gauging quality can then be combined for all fingertip ridge flow blocks and scaled to a meaningful value. One such scoring methodology results in a score from 0 to 100 where, for example, 100 represents the best possible image.
In
In
By way of illustration with reference to a real-world fingerprint, and with reference to
Once the correlation values have been determined for all windows in the fingerprint image, the resulting correlation map can be processed to identify the peak ridge flow angle (if any) found at every window within the image, thus producing the raw ridge flow map. This data can be further analyzed to remove any singularities and smoothed or averaged to generate an average ridge flow map as illustrated in
For example, and with reference to
In more detail, and with reference to
Small isolated clusters, such as 2020, are removed or are ignored and the larger clusters processed to determine the orientation of the cluster. The median angle from all clusters is then selected as the overall orientation angle for the hand. The clusters are then again processed to identify the groups that fall along the orientation angle with those clusters that fall on the hand orientation angle being grouped into fingers 2110 as seen in
One exemplary advantage of trimming the fingertip, finger, palm, or other target finger is to isolate only the image area directly associated with the target feature. For example, fingertips may be trimmed to length at the inter-digital crease. One exemplar method may use anthropomorphic rules to associate the finger width to the fingertip length. A second exemplar method may parse the ridge flow map to identify the breakdown in ridge flow commonly associated with the inter-digital crease.
Ultimately, the result is the identification of all ridge flow detections that make up the fingertip as well as the bounding box of the fingertip cluster. As with the other aspects of this invention, this can be implemented independent of input image resolution since the determination of ridge flow isolates the above determinations from resolution.
A non-limiting example is illustrated with reference to
As seen in
The following section describes the interim testing products developed as a part of the design and implementation process of the invention.
The first column of data is the DC term—the average pixel value for the sampled window. For the remaining columns the brightness of the pixel is related to the strength of the coefficient, i.e., the brighter the pixel the stronger the correlation between the sub-window of the image and the target sinusoid. Following each column from top to bottom there exists a trend in the bright peaks as they move around the fingertip center in response to the general trend of fingerprint ridge flow.
The described systems and methods can be implemented on an image processing device, fingerprint processing device, or the like, or on a separate programmed general purpose computer having image processing capabilities. Additionally, the systems and methods of this invention can be implemented on a special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit element(s), an ASIC or other integrated circuit, a digital signal processor, a hard-wired electronic or logic circuit such as discrete element circuit, a programmable logic device such as PLD, PLA, FPGA, PAL, or the like. In general, any device capable of implementing a state machine that is in turn capable of implementing the flowcharts illustrated herein can be used to implement the image processing system according to this invention.
Furthermore, the disclosed methods may be readily implemented in software using object or object-oriented software development environments that provide portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed system may be implemented partially or fully in hardware using standard logic circuits or a VLSI design. Whether software or hardware is used to implement the systems in accordance with this invention is dependent on the speed and/or efficiency requirements of the system, the particular function, and the particular software or hardware systems or microprocessor or microcomputer systems being utilized. The systems and methods illustrated herein however can be readily implemented in hardware and/or software using any known or later developed systems or structures, devices and/or software by those of ordinary skill in the applicable art from the functional description provided herein and with a general basic knowledge of the computer and image processing arts.
Moreover, the disclosed methods may be readily implemented in software executed on programmed general purpose computer, a special purpose computer, a microprocessor, or the like. The software can also be stored on a computer readable medium. In these instances, the systems and methods of this invention can be implemented as program embedded on personal computer such as JAVA® or CGI script, as a resource residing on a server or graphics workstation, as a routine embedded in a dedicated fingerprint processing system, as a plug-in, or the like. The system can also be implemented by physically incorporating the system and method into a software and/or hardware system, such as the hardware and software systems of an image processor.
It is, therefore, apparent that there has been provided, in accordance with the present invention, systems and methods for fingerprint quality and segmentation which may be particularly useful when used with fingerprint preview image(s). While this invention has been described in conjunction with a number of embodiments, it is evident that many alternatives, modifications and variations would be or are apparent to those of ordinary skill in the applicable arts. Accordingly, it is intended to embrace all such alternatives, modifications, equivalents and variations that are within the spirit and scope of this invention.
This application is a Continuation of U.S. patent application Ser. No. 13/848,908, filed Mar. 22, 2013, which is a Continuation of U.S. patent application Ser. No. 13/047,568, filed Mar. 14, 2011, now. U.S. Pat. No. 8,452,060, which is a Divisional of U.S. patent application Ser. No. 11/426,114, filed Jun. 23, 2006, now U.S. Pat. No. 7,936,907, which claims the benefit of and priority under 35 U.S.C. § 119(e) to U.S. Patent Application No. 60/795,215, filed Apr. 26, 2006, entitled “Real Time Fingerprint Quality and Segmentation,” each of which are incorporated herein by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| 60795215 | Apr 2006 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11426114 | Jun 2006 | US |
| Child | 13047568 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16411981 | May 2019 | US |
| Child | 16989069 | US | |
| Parent | 16111890 | Aug 2018 | US |
| Child | 16411981 | US | |
| Parent | 15720408 | Sep 2017 | US |
| Child | 16111890 | US | |
| Parent | 15485969 | Apr 2017 | US |
| Child | 15720408 | US | |
| Parent | 15204696 | Jul 2016 | US |
| Child | 15485969 | US | |
| Parent | 14873719 | Oct 2015 | US |
| Child | 15204696 | US | |
| Parent | 14628426 | Feb 2015 | US |
| Child | 14873719 | US | |
| Parent | 13848908 | Mar 2013 | US |
| Child | 14628426 | US | |
| Parent | 13047568 | Mar 2011 | US |
| Child | 13848908 | US |
