Patent Application
20200050849
The present invention provides an image processing method and apparatus for iris obfuscation.
The iris surrounds the dark, inner pupil region of an eye and extends concentrically to the white sclera of the eye.
A. K. Jain, A. Ross, and S. Prabhakar, “An introduction to biometric recognition,” IEEE Trans. Circuits Syst. Video Technol., vol. 14, 2004 discloses that the iris of the eye is a near-ideal biometric.
For the purposes of recognition, typically an image of an iris region is acquired in a dedicated imaging system that uses infra-red (IR) illumination with the eye aligned with the acquisition camera to bring out the main features of the underlying iris pattern.
An iris pattern is a gray-scale/luminance pattern evident within an iris region that can be processed to yield an iris code. The iris pattern can be defined in terms of polar co-ordinates and these are typically converted into rectangular coordinates prior to analysis to extract the underlying iris code.
An iris code is a binary sequence obtained after analysis of the iris pattern. A typical iris code contains 2048 bits. Note that some bits are effectively redundant, or ‘fragile’, as they are nearly always set to a ‘1’ or a ‘0’ as disclosed in K. Hollingsworth, K. W. Bowyer, and P. J. Flynn, “All Iris Code Bits are Not Created Equal,” 2007 First IEEE Int. Conf. Biometrics Theory, Appl. Syst., 2007. Some of these fragile bits can be predicted in advance and as they offer less differentiation, they are often ignored when determining a match.
Nonetheless, systems supporting the acquisition of iris data from mobile persons are known, for example, as disclosed in J. R. Matey, O. Naroditsky, K. Hanna, R. Kolczynski, D. J. Lolacono, S. Mangru, M. Tinker, T. M. Zappia, and W. Y. Zhao, “Iris on the Move: Acquisition of Images for Iris Recognition in Less Constrained Environments,” Proc. IEEE, vol. 94, 2006. This employs specialized lighting and requires people to walk along a specified path where multiple successive iris images are acquired under controlled lighting conditions. The system is proposed for airports where iris information is being used increasingly to verify passenger identity.
Separately, each of: C. Boyce, A. Ross, M. Monaco, L. Hornak, and X. L. X. Li, “Multispectral Iris Analysis: A Preliminary Study,” 2006 Conf. Comput. Vis. Pattern Recognit. Work., 2006; M. Vilaseca, R. Mercadal, J. Pujol, M. Arjona, M. de Lasarte, R. Huertas, M. Melgosa, and F. H. Imai, “Characterization of the human iris spectral reflectance with a multispectral imaging system.,” Appl. Opt., vol. 47, pp. 5622-5630, 2008; and Y. Gong, D. Zhang, P. Shi, and J. Yan, “Optimal wavelength band clustering for multispectral iris recognition,” Applied Optics, vol. 51. p. 4275, 2012 suggest that iris patterns from lighter color eyes can be adequately acquired, but that eyes of darker color are difficult to analyze using visible light,
H. Proenca and L. A. Alexandre, “Iris segmentation methodology for non-cooperative recognition,” IEE Proceedings—Vision, Image, and Signal Processing, vol. 153. p. 199, 2006; and A. E. Yahya and M. J. Nordin, “Non-cooperative iris recognition system: A review,” Inf. Technol. (ITSim), 2010 Int. Symp., vol. 1, 2010 disclose non-cooperative iris acquisition, typically obtained at a distance of 3-10 meters using directed IR sources.
As imaging subsystems on smartphones continue to improve in quality of acquisition and as image analysis and post-processing technique also continue to improve, a point at which the quality of images from conventional digital cameras and smart-phones becomes of sufficient quality to analyze to a sufficient degree to determine some of the underlying features of an iris pattern will be reached.
For example U.S. Pat. No. 7,697,735 discloses identifying a person from face and iris data from a single 5 megapixel image. U.S. Pat. No. 7,697,735 provides recommended minimum sizes for face and eye features to enable a sufficiently accurate degree of recognition. However it does not specify any details of lighting or acquisition conditions and most iris acquisitions would not be of sufficient accuracy in an unconstrained use case. Nevertheless we note that the latest handheld devices can feature imaging subsystems with up to 40 megapixel resolutions and high power IR LEDs can be used to improve acquisition lighting conditions.
Other techniques such as high dynamic range (HDR) imaging combine more than one digital image to provide a combined image with improved image quality. This is a standard feature on most smartphone imaging systems and typically two images are acquired in sequence and combined, post-acquisition, to provide a sharper and higher quality final image. Techniques are well known in the literature to combine more than one image and as acquisition systems achieve higher frame rates (currently 60-120 frames per second for preview but likely to double with next-generation technology) it will be practical to capture as many as 8-10 images within the same time window used today to acquire two images. Taking advantage of sub-pixel registration or super-resolution techniques will therefore provide images with significantly higher local image contrast and sharpness than today's devices provide.
Thus it highly likely that images acquired with the next generation of imaging devices will be of sufficient quality to enable the determination of iris patterns from faces in standard images. This makes normal personal portraits and small-group photos a potential source for personal iris patterns with a high risk of such biometric information being used for a range of criminal activities ranging from identity theft, forging of personal identity documents up to gaining access to facilities protected by biometric security measures.
US 2009/0141946, Kondo discloses detecting an iris region of an eye from an original image and performing image conversion on the detected iris region so that feature data unique to the person cannot be extracted. For example, the iris region is divided into a plurality of portions and respective images of divided portions are re-arranged in a predetermined order or at random.
US 2010/0046805, Connell discloses generating a cancelable biometric including shifting at least one pixel region in a biometric image comprised of pixel regions. The pixel region is combined with at least one other pixel region to form a replacement region for the at least one pixel region to form a transformed image. The biometric image is reused to generate another transformed image if the transformed image is to be canceled.
Accordingly to a first aspect of the present invention there is provided an image processing method as claimed in claim 1.
In a second aspect there is provided an image processing method as claimed in claim 2.
In a third aspect there is provided an image processing method as claimed in claim 3.
Embodiments of the invention (i) identify candidate iris regions within digital images; (ii) segment and analyze such regions to determine if they provide an iris pattern of sufficient quality to pose a risk of theft of the associated biometric; (iii) determine and calculate a suitable substitute biometric of similar appearance and aesthetic quality, and (iv) on storage, transmission or otherwise making permanent the original image data the at-risk iris patterns are substituted in the original image.
Other aspects of the invention provide a computer program product comprising a computer readable medium on which instructions are stored which when executed on an image processing device perform the steps of claims according to the claimed invention.
Various embodiments of the invention will now be described by way of example with reference to the accompanying drawings, in which:
Referring now to
Face detection in real-time has become a standard feature on most digital imaging devices, for example, as disclosed in WO2008/018887 (Reference: FN-143). Further, most cameras and smartphones also support the real-time detection of various facial features and can identify specific patterns such as ‘eye-blink’ and ‘smile’ so that for example, the timing of main image acquisition can be adjusted to ensure subjects within a scene are in-focus, not blinking or are smiling such as disclosed in WO2007/106117 (Reference: FN-149). Other approaches substitute in-focus, non-blinking or smiling portions of preview images into corresponding out-of-focus, blinking or non-smiling portions of main images to improve image quality for example as disclosed in WO2008/150285 (Reference: FN-172). Where such functionality is available in an image processing device, detecting and tracking face regions and eye regions within those face regions imposes no additional overhead and so this information is available continuously for an image stream.
In the present embodiment, a face/eye tracking subsystem 16 locates and tracks face regions within an image stream. However, it will be appreciated that a face/eye detection sub-system could simply be applied to single still images to detect any, face, and eye regions within the image.
In any case, the face/eye sub-system 16 acquires either a still image or an image from a stream, step 28, and then locates eye-regions within any detected face regions in the image, step 30,
Preview images can be displayed in a display 18 and in some cases tracked face or eye regions can be indicated in the preview display.
The face/eye tracking subsystem 16 thus detects candidate eye regions and potential ‘at risk’ candidates can be flagged as the image or image stream is being processed. As indicated, the quality criteria used by the face/eye tracking subsystem 16 at step 32 can be quite rudimentary and additional, more detailed analysis can be made at the time when an acquisition is completed and an image (or image sequence) is (being) committed to storage 22 or transmitted beyond the device over a network connection (not shown).
Once a user initiates an action that will commit an image to permanent or semi-permanent storage 22, the sub-system 16 initiates the check to determine if the image (or image sequence) contains ‘at risk’ eye regions. If none are present then the image is saved normally. However if the image meets criteria for the above parameters, then ‘at risk’ regions are present and these eye regions may contain iris regions which may need to be substituted as described in more detail below. In this case, they are passed by the sub-system 16 to an iris analysis and processing sub-system 20 at step 34.
Firstly, at step 36, the iris regions are extracted from ‘at risk’ eye regions and a more detailed analysis performed to confirm if a valid iris pattern is detectable. The iris region can be determined by edge analysis or Hough-transform. J. Daugman, “New methods in iris recognition,” IEEE Trans. Syst. Man. Cybern. B. Cybern., vol. 37, pp. 1167-1175, 2007 discloses a range of additional refinements which can be utilized to determine the exact shape of iris and the eye-pupil. It is also common practice to transform the iris from a polar to rectangular co-ordinate system, although this is not necessary. The end result of this process is an iris region separated from the main image with a secondary inner-boundary corresponding to the iris/pupil boundary of the eye. This approximately doughnut-shaped region provides the input for the next stage of iris analysis.
Embodiments of the present invention can employ combinations of the following criteria to confirm if extracted iris regions are at risk of providing a pattern which enables recognition:
It should be noted that each of the above quality measures can be determined on a real-time basis within a current state-of-art digital imaging device. Other schemes for assessing iris quality are provided in:
If the designated criteria for an iris region are met, then an iris pattern is provided for the iris region; as well the color of the iris region.
In some embodiments the iris pattern may be evaluated and compared against a set of known patterns—e.g. the owner of the device and perhaps family members and friends. Certain actions may be pre-programmed according to the identified person, for example, the device may also signal the user of the device that ‘at risk’ iris patterns have been identified and are being substituted, step 38.
In the embodiment of
At step 40, the BAU extracts the relevant iris code from the detected iris pattern and records this temporarily either in secure memory 27 or system memory 29. Where a BAU is available, this step are used as an additional test of the quality of detected ‘at risk’ iris regions. Thus, if an ‘at risk’ region is rejected by the BAU, step 42, then an error code from the BAU can verify that certain quality metrics are not met or that other aspects of the region prevent a useful biometric being extracted.
Nonetheless, it may be possible to correct the iris region, step 44, for example, with an alternative contrast enhancement such as outlined above, and to then re-submit the iris region for BAU analysis—this may involve again checking the enhanced iris against known patterns at step 38. This loop may be performed iteratively until all possible error corrections have been attempted.
If error correction is not possible or exhausted, step 46, the iris region is re-marked as not being at risk.
Where a BAU is not available, as in
In any case, if an iris code can be extracted from the iris region, either with or without a BAU, step 48, the iris is submitted for further processing, step 50 where a replacement iris pattern and ultimately a replacement iris region is provided.
Before continuing, it should be noted that a unique replacement iris need not be required for every image. In some embodiments, a new replacement iris is only provided when a new iris pattern is identified, for example in step 38. Thus where a device keeps a local information dataset for a group of persons that are regularly photographed or videoed by the user of the device, then each person can have a unique replacement iris pair, possibly stored in secure memory 27, which is used whenever they are identified in an image. Another set of replacement iris patterns can be used for unidentified persons. In such embodiments, a device only needs to occasionally obtain, or generate a set of replacement iris patterns. This may be implemented in-camera, but equally these may be obtained via a secure network service or a specialized app running on the device. Embodiments of the invention attempt to generate or obtain a natural looking iris to substitute for the original iris detected in an acquired image as described in
Providing and substituting an iris may be achieved in a number of ways. Referring now to
Referring now to
The standard iris images can be supplied with the iris analysis and processing software 20 and so can be common to all devices using the technique; or the library can be built up on the device itself either from images acquired by the device; or through the device acquiring images from a network source, such as the Internet.
For a given input eye region acquired from an image such as described above in relation to
Ideally, each standard iris image comprises a complete iris and pupil, for example, as shown in the standard iris image 600 of
As before for
It will be appreciated that the outer boundary 704 of the input iris may not be circular where the iris is occluded by an eye lid and also the area of the input iris may not be the same as the area of the selected standard iris.
The present implementation is based on replacing the details of the input iris using the details from the standard iris.
These details are determined on a layer-by-layer basis, with for example k=4 layers, by successively blurring each of the input iris image and the standard image as follows: In one example, the blurring is performed by box filtering with a k*k kernel where k=[1, 2, 4 and 8] % of the length of the crop 708. (It will be appreciated that if scaling were performed after blurring, then pre-blurred standard image information could be employed.)
Thus, for each of the standard iris and the input iris, the image IRIS is blurred to provide an image irisBlurred. Each irisBlurred image is then successively blurred for k=2 to 4 as follows:
irisBlurred_1=Filter(IRIS,k[1])
for i=2:4 irisBlurred_i=Filter(irisBlurred(i−1),k[i])
Then, for each image IRIS and for each layer, detail layers are extracted by subtracting the blurred images from the previous image as follows:
detail_1=IRIS−irisBlurred_1
for i=2:4 detail_i=irisBlurred_(i−1)−irisBlurred_i
The non-iris areas outside the iris boundaries and the pupils, can be removed (blanked) from each of the 2×4 detail images using masks based on the maps 602 and 702. In some cases, the masks can be slightly blurred using, for example, an m*m kernel, wherein m=2% of crop length box filter to provide for better transitions in the final image.
The iris details of the original iris image crop 708 in
irisBase=irisIN−Σi=14detail_i
where detail_i are the iris portions of the detail images calculated from the input iris image, irisIN.
Now the details removed from original image can be replaced with the details for the standard iris as follows:
irisOUT=irisBase+0.75*Σi=14detail_i
where detail_i are the iris portions of the detail images calculated from the standard iris image.
It will be appreciated that where the iris boundaries and the pupil locations of the input eye region iris and the standard iris do not closely correlate, an affine transformation based on the maps 602 and 702 can be applied when adding the detail layers for the standard iris to the irisBase image to produce irisOUT.
Referring back to
If the codes are sufficiently distinct, the embodiment then continues by generating a replacement iris region based on the replacement iris pattern and re-constructing the iris region within the original image based on the replacement iris region, step 68. This step includes matching and blending the luminance, color and any specular reflections or eye glint from the original ‘at risk’ region so that the replacement eye region presents a substantially similar appearance. This will be described in more detail below, but as will be appreciated, once this step is complete, the image and/or iris region within the image can be marked as secure, step 70, and the method can proceed to process any further ‘at risk’ iris patterns identified within the image by returning to step 58.
Where a replacement iris pattern has previously been generated for an iris pattern recognized within an image, steps 58-66 can be skipped and the previously generated replacement iris pattern simply retrieved from memory before continuing with steps 68 and 70.
As mentioned above, conventional iris based BAU typically use a gray-scale iris pattern as a starting point. This practice originates from the use of IR illumination to enhance the iris pattern and the consequent single-channel image data obtained (gray-scale).
If an authentication system employs a color check in addition to a BAU, then an additional feature is that stored iris patterns used in in the approach of
In such an embodiment the iris patterns of friends and family can be used to generate replacement patterns for each other.
In another alternative to the approach of
This technique has been employed for synthesizing irises, for example, as disclosed in Z. Wei, T. Tan, and Z. Sun, “Synthesis of large realistic iris databases using patch-based sampling,” 2008 19th Int. Conf. Pattern Recognit., no. 1, pp. 1-4, December 2008. Also, L. Liang, C. Liu, Y.-Q. Xu, B. Guo, and H.-Y. Shum, “Real-time texture synthesis by patch-based sampling,” ACM Transactions on Graphics, vol. 20. pp. 127-150, 2001 discloses patch-based sampling to scramble a known iris pattern while retaining a realistic looking eye region.
In another alternative to the embodiment of
In another alternative, especially useful where a BAU is not available, again instead of steps 58-66, the iris code for the original iris is scrambled and used as a basis for reconstructing a replacement iris pattern. (Note that because an iris code is typically derived from a lossy transformation of the original iris pattern, there is a one-to-many relationship between an iris code and corresponding irises.) In this approach, the iris code of the ‘at risk’ region is determined. A range of bits of this code are then ‘flipped’; typically of the order of 50% of bits are changed, but the exact number and relative locations of bits may be randomized. For example, bits known to be fragile might not be flipped as these are often masked by BAUs when comparing iris codes.
The remainder of this approach is based on the work described in S. Venugopalan and M. Savvides, “How to Generate Spoofed Irises from an Iris Code Template,” IEEE Trans. Inf. Forensics Secur., vol. 6, pp. 385-395, 2011. Here, a unique discriminating pattern is next determined from the ‘flipped’ code (an anti-code for that of the original iris pattern) and a replacement iris pattern is generated on a neutral iris template. By flipping more than 50% of bits in the underlying iris code, a large Hamming Distance is ensured and thus cross-checking by a BAU is not required.
Another approach to constructing the replacement iris pattern of steps 58-62 is based on J. Galbally, A. Ross, M. Gomez-Barrero, J. Fierrez, and J. Ortega-Garcia, “Iris image reconstruction from binary templates: An efficient probabilistic approach based on genetic algorithms,” Comput. Vis. Image Underst., vol. 117, pp. 1512-1525, 2013. Due to the computational complexity of these techniques, the replacement iris may need to be determined outside the image capture device—e.g. as a secure network service.
In still further embodiments, instead of steps 58-62, a synthesized, artificial or random iris pattern is generated using techniques described in, for example, S. Shah and A. Ross, “Generating Synthetic Irises by Feature Agglomeration,” 2006 Int. Conf. Image Process., 2006; L. Wecker, F. Samavati, and M. Gavrilova, “A multiresolution approach to iris synthesis,” Comput. Graph., vol. 34, pp. 468-478, 2010; or L. Cardoso, A. Barbosa, F. Silva, A. M. G. Pinheiro, and H. Proenca, “Iris Biometrics: Synthesis of Degraded Ocular Images,” vol. 8, no. 7, pp. 1115-1125, 2013; or other methods such as are reviewed in Venugopalan et al referred to above.
Turning now to
Again, an iris region with a corresponding code which has not been recognized previously are provided, step 72. As in the alternative described above, a selected number of bits of the iris code are flipped, step 74. An iris pattern (DP1) is synthesized based on the flipped iris code, step 76, and an iris synthesized from the pattern DP1, step 76. The synthesized replacement iris is sent to a BAU, step 78 where it is analyzed, step 80.
If the BAU detects an error in the synthesized iris, an error is returned, step 82. There may be a possible fix, step 84, but if all fixes are exhausted and no suitable discriminating iris can be generated, the user is notified, step 86 and the process continues to step 72 and the next iris in the image for processing.
Otherwise, the BAU provides the iris code for the synthesized iris (this should correspond with the flipped code), step 88. The Hamming Distance between the respective synthesized and original iris codes can be determined, step 90. Again, in some embodiments of the invention, the threshold for Hamming Distance could be user selected within the range 0.33 to 0.5 or it could be a function of a user's chosen security settings, so that a higher HD would be employed for more security conscious users.
If the HD is suitably distinct, the process proceeds, step 92, by substituting the synthesized iris for the original iris as in step 70 of
In relation to the iris substitution performed in each of steps 70 and 92, it will be understand that a replacement and original iris may not be identical in size/shape and it can be necessary to blend the replacement iris into the original acquired image. In addition it is important to match the overall luminance and color of the original and replacement regions so that the replacement iris appears as natural as possible.
In one embodiment this substitution involves the following steps:
Embodiments of the present invention are particularly suitable for images in a color space where there is separation between intensity/luminance and chrominance, e.g. YCC or LAB, where one image plane, in these cases, Y or L provides a greyscale luminance component. In such cases, it is the Y or L plane of the iris region of an image which is replaced with the Y or L plane information from another iris.
In some cases, some matching of the luminance histograms to keep the replacement iris region at the same brightness level can be performed.
The present application is a continuation of U.S. application Ser. No. 15/301,937 filed on Oct. 4, 2016 which is a National Stage Entry of International Application No. PCT/EP2015/056815, filed Mar. 27, 2015 which claims priority to U.S. Application No. 61/975,262 filed Apr. 4, 2014.
| Number | Date | Country | |
|---|---|---|---|
| 61975262 | Apr 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15301937 | Oct 2016 | US |
| Child | 16656303 | US |
