Patent Application
20240071142
The present invention relates to the detection of moiré patterns in digital images. Furthermore, the present invention relates to the liveness detection of captured biometric features for biometric verification.
Moiré patterns are considered to be interference patterns occurring due to the overlap of the digital grids of the camera sensor resulting in a high-frequency noise in the image. Detection and removal of these patterns is a crucial step in many applications. For example, detection and removal of moiré patterns is an important step in a biometric verification system, where these patterns may be used to determine liveness of the captured biometric features, thereby preventing spoofing attacks. Similarly, moiré patterns may be used in scanning applications to improve Optical Character Recognition (OCR).
An example of a known method for detecting moiré patterns in a digital image is presented in E. Abraham, “Moiré Pattern Detection using Wavelet Decomposition and Convolutional Neural Network,” 2018 IEEE Symposium Series on Computational Intelligence (SSCI), Bangalore, India, 2018, pp. 1275-1279, doi: 10.1109/SSCI.2018.8628746. A disadvantage of the current techniques for detecting moiré patterns is that their detection accuracy may be affected by the intensity profile of the digital image. This is because the frequency strength of the moiré patterns, which determines their visibility in the captured image, highly depends on the pixel intensity of the digital image. For example, the lower the pixel intensity of the captured, thereby the darker the captured image, the lower the frequency strength of the moiré patterns present in the image. Similar effects may be observed in overexposed areas of the captured image. As a result, due to the pixel intensity variations in a digital image, the moiré patterns may not be accurately detected and/or extracted, which may be detrimental in preventing spoofing attacks in a biometric verification system.
An aim of the present invention is to provide a system and a method for detecting moiré patterns in a digital image that overcomes the disadvantages of the prior art solutions.
A further aim of the present invention is to provide a system and a method for verifying liveness of captured biometric features of a person in a biometric verification system.
The aims of the present invention are achieved according to the invention with the system and method showing the technical characteristics of the respective independent claims. Preferred embodiments of the present invention are disclosed in the dependent claims.
According to an aspect of the present invention, a method for detecting moiré pattern information in digital images is presented. The method comprises the steps of:
The method of the present invention enhances the detection of moiré patterns in a digital image by processing and combining moiré pattern information detected from a plurality of digital images of the same subject and/or scene captured at different image resolutions and/or exposures. In general, the pixel intensity of a captured object and/or scene would vary within the captured digital image, resulting in image sections having different light/pixel intensity values. As such, regions of the digital image may be presented with different light/pixel intensity values. Moiré patterns are dependent on the pixel intensity of the digital image. For example, moiré patterns may be difficult to detect in dark or overexposed regions of a captured image. Therefore, with traditional techniques, only a small portion of the moiré patterns may be detected in a digital image, due to camera sensor limitations and lighting conditions. Enhancing detection of moiré patterns from regions of the digital image having different pixel intensity values is important in a range of applications which include but not limited to biometric verification, Optical Character Recognition (OCR), and the like. The method of the present invention overcomes the disadvantages of the prior art solutions by extracting moiré pattern information from digital images of the same subject and/or scene captured at different image resolution and/or exposure. As such, it is possible to detect moiré patterns from different regions of the digital images. For example, by progressively increasing the image exposure between successive digital images, it is possible to increase the light intensity of darker regions of the digital images, which may result in the exposure of moiré patterns, leading to their accurate detection. Similarly, by varying the image resolution between successive digital images, it is possible to change the moiré patterns captured in the digital image. As such, different moiré patterns may emerge depending on the image resolution, which may provide further enhance the detection of the moiré patterns from the captured image and/or scene In the method presented herein, moiré pattern information detected and/or extracted from the set of digital images is combined into a single digital image for the representation of the captured object and/or scene. The combined digital image may be generated by combining details from different captured digital images in the set that contain moiré patterns. For example, the merged digital image may be generated using a high dynamic range imaging technique known in the art, which involves capturing multiple images of the same scene using different exposure values, and then combining those images that contained moiré patterns into a single image that represents the range of tonal values within the captured scene and/or object. As such, the merged digital image would contain moiré pattern information from different areas of the digital image, thereby overcoming the disadvantages of the prior art solutions.
According to embodiments of the present invention, the detection of Moiré patterns may be performed using wavelet decomposition and/or a multi-input deep Convolutional Neural Network (CNN).
According to embodiments of the present invention, processing of the set of digital images comprises aligning the subject and/or scene captured in each of the digital images in the set. In general, to compensate for any movement of the subject and/or scene occurred between successive digital images, e.g. head movement of the captured person, the digital images are aligned. For example, the alignment of the digital images may be performed using known image registration techniques known in the art, such as processing the digital images using a digital image aligning algorithm, or another known techniques. For example, image registration may be based on feature registration, pixel-based alignment, or any other known technique.
According to embodiments of the present invention, processing the digital images comprises segmenting each digital image into a grid of predetermined dimensions. According to embodiments of the present invention, processing of the digital images comprises detecting in each digital image the grid sections comprising moiré pattern information. Segmentation aids the temporal analysis of the digital images in the set, thereby simplifying the detection of similar regions in the set of digital images. As such, sections of the digital images containing moiré pattern information may be easily detected and compared to corresponding sections of the remaining digital images. It should be noted that the segmentation of an image may be performed in different ways. For example, the image analysis may grid-based, where the digital image is divided into subsections, or it may object segmentation based, where the digital images are analysed to detect similar regions, e.g. the face of a person.
According to embodiments of the present invention, processing of the digital images comprises analysing the luminous intensity of each digital image in the stack both in spatial and temporal direction to determine a combined intensity profile for the set of digital images. Based on the segmentation, the pixel intensity of each digital image is analysed both in spatial and temporal direction to obtain the intensity profile of the stacked digital images. The intensity profile information may be used to determine the frequency strength and/or pattern of the moiré interference noise detected in each digital image and correlate them with the corresponding image exposure and/or resolution in the stack of digital images. The correlation of Moiré patterns may be performed using Feature detection and Matching methods like Fourier/Wavelet Transform, SURF, SIFT features, and the like. Also, the correlation may be performed using trained Deep Learning networks like Convolution Neural Networks (CNN).
According to embodiments of the present invention, generating the merged digital image comprises combining details from different captured digital images in the set that contain moiré patterns into a single image that represents the range of tonal values within the captured scene and/or object. Further, generating of the merged digital image may comprise spatially correlating the extracted moiré pattern information detected from each digital image in the set. According to embodiments of the present invention, generating of the merged digital image comprises extracting a frequency profile of the moiré pattern information. The moiré patterns detected and/or extracted from each digital image may be mapped on the segmentation grid, thereby identifying the sections of the digital image containing moiré pattern information. Due to the variation in the exposure and/or resolution of the digital images in the set, the frequency strength of the moiré patterns detected in each digital image may be different from one another. The merged digital image may be generated based on the selection and combination of digital images and/or sections of digital images. For example, the digital images and/or sections of digital images may be selected based on the detected moiré pattern information meeting certain criteria, e.g. the frequency strength being within a specific range. As such, the extracted moiré patterns detected and/or extracted from the selected sections may be spatially correlated to determine the distribution profile of the resulting moiré frequency in the merged digital image. Furthermore, other known methods may be used to generate the merged digital image. For example, the merged image may be generated by selecting the max frequency or combining the low-high frequencies of the detected moiré patterns together in the temporal direction in the stack of digital images.
According to embodiments of the present invention, the merged digital image is a high dynamic range image.
According to a second aspect of the present invention, a method for determining liveness of biometric features of a person is presented. The method comprises the steps of: capturing a set of digital images of one or biometric features of a person, the set of digital images comprising images captured by one or more cameras and at different resolutions and/or exposure;
According to embodiments of the second aspect of the present invention, determining liveness comprises the step of extracting a moiré frequency strength from the moiré pattern information and comparing the extracted moiré frequency strength with a liveness threshold.
According to embodiments of the second aspect of the present invention, if the extracted moiré frequency strength is within a first range from the liveness threshold, validating the digital image of the one or more biometric features, otherwise rejecting the digital image.
The method of the present invention may be used in a range of applications. For example, the method for detecting moiré patterns in digital images may be part of a biometric verification system. In a biometric verification system, it is important to be able to distinguish between a spoofing attack and a legitimate access request form a registered user. In spoofing attacks, e.g. facial spoofing, a fraudulent user may try to gain illegitimate access by using a photo, video, or other material of an authorised user's face. As such, the biometric verification system needs to be able to assess the liveness of the captured biometric features accurately. Moiré patterns may be used to determine liveness of the captured biometric features by assessing their resulting frequency and/or pattern profile. For example, capturing an object from a photograph, or video would result in different moiré patterns than capturing the same object in the real world. However, since moiré patterns depend on the intensity profile of the captured digital image, the biometric verification systems may incorrectly detect the moiré patterns from a photograph and/or video of a registered user, thereby leading to a successful spoofing attack. Similarly, a biometric identification system may reject legitimate biometric verification request from a registered user due to the incorrect detection of the Moiré pattern from a live captured digital image. For example, in images with lower intensity, i.e. darker images, the moiré patterns may not be visible, or their frequency may low, and as such moiré patterns may not be accurately detected by the biometric verification system resulting in a successful spoofing attack. The present invention overcomes this issue by extracting a profile of the moiré patterns detected from a plurality of images of the same subject that have been captured at different resolution and/or exposure. As such, with the present invention, it is possible to accurately extract the profile and/or frequency of the moiré patterns from a digital image, thereby improving the accuracy of the biometric verification systems.
According to embodiments of the second aspect of the present invention, capturing of the digital images comprises varying the resolution and/or exposure of the at least one camera within a predefined range between successive capturing of digital images of the one or more biometric features. According to embodiments of the second aspect of the present invention, the resolution and/or exposure of the camera is varied by a predetermined value for each digital image in the set. For example, the exposure and/or image resolution may be progressively adjusted in a stepwise process, wherein each adjustment step is of a predetermined value. For example, the exposure of each digital image may be varied by adjusting the shutter speed and/or aperture of the camera. Similarly, the image resolution may be adjusted by changing the resolution of the camera sensors.
According to embodiments of the second aspect of the present invention, the one or more biometric features are facial features.
According to a third aspect of the present invention, a liveness detection system for biometric authentication is presented, the system comprising:
The following drawings are provided as an example to explain further and describe various aspects of the invention:
The present invention will be illustrated using the exemplified embodiments shown in
Detection of Moiré patterns in digital images may be used in a range of applications. For example, detection of moiré patterns may aid the detection of the liveness of biometric features in a biometric verification system. Similarly, detection of moiré patterns may be used in an OCR application to improve the detection of characters from a scanned document or a photo. In particular, with regards to the liveness detection, moiré pattern detection in the captured digital images has been shown to improve the liveness detection accuracy. It is known that moiré patterns detected in a captured digital image of a live subject, e.g. a person sitting in front of a camera, would be different from the moiré patterns detected when the captured image is a scan of a document or a photograph of the same subject. As shown in
Traditionally liveness detection may be performed by detecting moiré patterns on the captured image. However, a disadvantage of the known techniques is that they rely on extracting moiré patterns from a single image. It is well known that the accuracy of moiré pattern detection may be affected by the pixel intensity of the captured image. For example, it has been found that the lower the pixel intensity, and thus the darker the captured image, the lower the frequency strength of the moiré patterns. As such, the accuracy of the liveness detection may be highly affected by the pixel intensity of the captured image. Fluctuations in the accuracy of the liveness detection may lead to false positives, e.g. successful spoofing attacks, or false-negative, e.g. incorrectly rejecting live biometric features of a user. In contrast, the liveness detection system of the present invention aims to improve the liveness detection accuracy by assessing the moiré patterns detected from a set of multi-variate digital images taken at different image resolutions and/or exposures. In other words, the present invention varies the image resolution and/or exposure of the captured digital images to adjust the pixel intensity of the captured biometric features and/or the moiré patterns generated. By adjusting the light intensity and/or resolution of a captured image it is possible to adjust the frequency strength and/or profile of the detected and/or extracted moiré patterns, thus enabling detection of moiré patterns from captured images having varying pixel intensity.
Returning to
In different applications, such an OCR application, the processing pipeline may be adapted accordingly so that it only applies to the detection of the moiré patterns in a merged digital image, such as the one generated by the fused image of moiré patterns module 128. Similarly, a more simplified process pipeline may be adopted, that may include the steps of:
In general, the routines executed to implement the embodiments of the invention, whether implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions, or even a subset thereof, may be referred to herein as “computer program code,” or simply “program code.” Program code typically comprises computer-readable instructions that are resident at various times in various memory and storage devices in the computer and that, when read and executed by one or more processors in a computer, cause that computer to perform the operations necessary to execute operations and/or elements embodying the various aspects of the embodiments of the invention. The computer-readable program instructions for carrying out operations of the embodiments of the invention may be, for example, assembly language or either source code or object code is written in any combination of one or more programming languages.
The program code embodied in any of the applications/modules described herein is capable of being individually or collectively distributed as a program product in a variety of different forms. In particular, the program code may be distributed using the computer-readable storage medium having the computer-readable program instructions thereon for causing a processor to carry out aspects of the embodiments of the invention. Computer-readable storage media, which is inherently non-transitory, may include volatile and non-volatile, and removable and non-removable tangible media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other robust state memory technology, portable compact disc read-only memory (CD-ROM), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and which can be read by a computer. A computer-readable storage medium should not be construed as transitory signals per se (e.g., radio waves or other propagating electromagnetic waves, electromagnetic waves propagating through a transmission media such as a waveguide, or electrical signals transmitted through a wire). Computer-readable program instructions may be downloaded to a computer, another type of programmable data processing apparatus, or another device from a computer-readable storage medium or an external computer or external storage device via a network. Computer-readable program instructions stored in a computer-readable medium may be used to direct a computer, other types of programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions that implement the functions/acts specified in the flowcharts, sequence diagrams, and/or block diagrams. The computer program instructions may be provided to one or more processors of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the one or more processors, cause a series of computations to be performed to implement the functions and/or acts specified in the flowcharts, sequence diagrams, and/or block diagrams.
In certain alternative embodiments, the functions and/or acts specified in the flowcharts, sequence diagrams, and/or block diagrams may be re-ordered, processed serially, and/or processed concurrently without departing from the scope of the invention. Moreover, any of the flowcharts, sequence diagrams, and/or block diagrams may include more or fewer blocks than those illustrated consistent with embodiments of the invention.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms “comprise” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Furthermore, to the extent that the terms “includes”, “having”, “has”, “with”, “comprised of”, or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising”.
While a description of various embodiments has illustrated all of the inventions and while these embodiments have been described in considerable detail, it is not the intention of the Applicants to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. The invention in its broader aspects is therefore not limited to the specific details, representative apparatus and method, and illustrative examples shown and described. Accordingly, departures may be made from such details without departing from the spirit or scope of the Applicants general inventive concept.
| Number | Date | Country | Kind |
|---|---|---|---|
| 21305016.4 | Jan 2021 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/086391 | 12/17/2021 | WO |
