Patent Application
20220358575
Not Applicable.
Not Applicable.
Many technologies have been developed to provide businesses and consumers with the ability to capture and transmit document images electronically via desktop and mobile devices. These remote deposit capture (RDC) technologies allow users to transmit instruments such as a check by sending images acquired from a digital camera, scanner, mobile phone, or other device in a matter of seconds. Users can take pictures of documents using the camera in such devices and then transmit the document images for further processing, such as submission for deposit into an account. These technologies can save money for institutions by reducing item processing costs and labor expense, and can provide substantial convenience for businesses and individuals.
Successful RDC processes begin with and depend upon capture of quality images of documents that adequately convey the substance of the information on the face of the documents. Various metrics that represent objectively measurable check image quality defects may be employed to assess the quality of candidate images for capture. With these metrics, deviations from a perfectly complete and accurate image can be assessed by standard technical measurements that do not involve subjective judgments.
Current technologies generally apply image quality metrics to the characteristics of the document image that appears in the foreground of the field of view of a camera. The image quality assessment hierarchy typically proceeds through defect assessment as applied to that foreground image, which may include capture system characteristics and calibration, image defect presence, initial data presence metrics, and various usability assessments.
The present invention presents methods and systems that expand image quality analyses beyond the document image in the foreground of the field of view of a camera associated with a mobile device. The invention relies upon image quality metrics applied to the entire image within the field of view of a camera combined with non-image metrics to provide a hybrid approach to capturing a quality image. The objects of this invention include: providing a method of capturing document images with a mobile device that is not dependent upon monitoring the image of the document; providing a method of capturing document images that is based upon the total image within the field of view of a camera satisfying a plurality of criteria; providing a method of capturing document images with a mobile device that does not rely upon feedback instructions to the user or satisfaction of alignment criteria; providing a method of capturing document images with a mobile device that includes at least one non-image quality criterion; and minimizing the need for user approval of multiple check images.
Other objects and advantages of the invention will be apparent from the following summary and detailed description of the invention, taken with the accompanying drawings.
The present invention is described in detail with reference to the following figures. The drawings are provided for purposes of illustration only and merely to depict example embodiments of the invention. These drawings are for illustrative purposes and are not necessarily drawn to scale.
The present invention is a system and method that may be carried out as a software application running on a device such as a mobile phone as part of an RDC system for capturing images of documents and transmitting them electronically to a remotely located financial institution, deposit processing service, or other target location. Mobile phones that use the application typically have a digital camera embedded in the device or otherwise communicatively coupled to the device's processor that may be used to capture images. The document images that are captured constitute the foreground portion of a composite image presented in the field of view of the camera. The invention provides a method and system for capturing such images that do not depend upon monitoring document images relative to one or more criteria and that are not dependent upon the document images satisfying monitoring or alignment criteria. Instead, the invention relies upon device orientation and the quality of the composite image that comprises a total image presented to a user within a camera's field of view. If the instrument is a check, remote deposit may proceed without user approval of an image of the backside of the check. Feedback regarding image quality may be provided to the user, but is not necessary. Similarly, an alignment guide may be provided to aid the user, but also is not necessary.
The present invention uses on-device software to allow image capture only when certain image and non-image criteria are satisfied. The analysis and image capture control are provided without the need for intermediate communication with a server. This results in increased speed of processing and reduced rates of rejection of images received for processing from the remote device.
In one embodiment, when a user activates the application on the device to make a financial transaction, the user may be provided initially with a form of welcome screen that may provide an option of making a deposit. Upon selecting such option, the user may then be provided with any of a variety of instructions, which may include, by way of example only, verification of payee, confirmation of consistency of amounts indicated by legal amount and courtesy amount, verification of signature, or instructions regarding the capturing of a check image.
The user may then be provided access to image capture capabilities. This may first include an invitation or instruction to capture an image of one or both sides of the check. Optionally, it may also include a request for entry of certain initial information such as, for example, the selection of a target account for the deposit.
The user may be given the option of operating in automatic or manual mode. By way of example, such options may be provided by on-screen “buttons” for automatic 240 and manual 245. Should the user select manual 245, the software provides for manual capture and the user may capture the image by, for example, touching a specified area of the screen, but only if the applicable image and non-image criteria are satisfied. If these criteria are not satisfied, the user will not be able to capture an image with the camera. In the automatic mode 240, the device will automatically capture the image 235 appearing in the field of view of the camera as shown in the viewfinder 230 when the composite image and device satisfy the applicable criteria over a specified period of time.
As shown in
Defect assessment involves both a quantitative measurement of an attribute of the image 435 (e.g., image luminosity) and a qualitative assertion about the presence of a defect (e.g., image too dark) applied to that image. A variety of quantitative image measurements are available and may be used. The corresponding qualitative assertions depend upon the thresholds that are established for each metric or combination of metrics that strike an appropriate balance between correctly identifying defects that may affect usability, and thus avoiding capture of bad images, and incorrectly identifying defects that won't affect the usability of the image. The quality of the performance of the system may be dependent upon setting optimum numerical thresholds based upon data concerning the frequency with which each defect occurs in the real world of image capture.
By way of example, one image quality criterion that may be applied is the average luminosity or “brightness” of the pixels within the image 435. As camera frames are received from the camera of the mobile device 110, the “red-green-blue” (RGB) values of the pixels of each image may be converted to grayscale and the luminosity value of each pixel determined using a specified formula, such as the NTSC formula: 0.299·Red+0.587·Green+0.114·Blue, where Red, Green, and Blue are the respective red, green, and blue values of the pixel. The calculated luminosity values of the pixels may be averaged over the entire image, with the result tested against a pre-determined threshold (for example, 0.2), where a lower value indicates a dark frame. If the mean value is less than the threshold value, then the luminosity criterion for the composite image has not been met.
By way of further example, another image criterion that may be applied is the focus quality of the image 435. Focus may be quantitatively assessed using a variety of measures, such as gradient measure, frequency domain, auto-correlation, or variance measure. Gradient measure is a common measure and involves calculating the sum of the difference between every nth pixel in both the X and Y directions of the image 435. As an image comes into focus, edges become sharper, thus increasing the gradient measure of the image. For example, as image frames are received from the camera of a mobile device, the pixels may be converted to grayscale and measured for a “focus score.” A focus score is a measure as a ratio of maximum video gradient between adjacent pixels, measured over the entire image and normalized with respect to the image's gray level dynamic range and pixel pitch. The following formula may be used to compute a score for image focus:
Other criteria may be applied. For example, and without limitation, the composite image 435 could be assessed as to whether it satisfies range of color values, depth, or distance criteria.
In addition to monitoring and evaluating the image frames from the camera's field of view, the process relies upon monitoring at least one non-image criterion. For example, the software may monitor the orientation of the device 110 to assure that the device is oriented in an acceptable direction before the image 435 shown in the viewfinder 230 is captured. This may be done by employing a gyroscope, which typically is present in mobile devices running Apple iOS or Google Android operating systems. The orientation monitoring process may entail evaluating whether the device is oriented in a sufficiently downward direction, within a pre-determined tolerance. By way of example, such a tolerance may be 0.85, where 1.0 represents gravity, but any tolerance may be used depending upon the orientation required.
A temporal element may be applied to assure stability of criteria satisfaction. For example, in automatic mode, upon satisfaction of the required criteria, picture taking may be initially deferred (e.g., 500 milliseconds) to assure such stability. After such deferral, image capture may be “scheduled,” and a countdown timer, such as 3-2-1, displayed on the device display may be used as a lead up to the action of image capture. If, during the countdown, the composite image 435 or the device 110 fails to satisfy any of the criteria, the countdown may abort without image capture.
Upon satisfaction of the criteria over time, the device may communicate in some manner with the user and image capture may occur either manually or automatically. For example, in automatic mode, after a plurality of composite image criteria are satisfied and one non-image criterion is satisfied over a specified time period, a countdown may appear on the video display screen with a message such as “Hold Steady,” followed by automatic capture of the image. In the manual mode, a signal may be provided to the user when the criteria are satisfied and the user may capture the image at any time thereafter so long as the criteria remain satisfied. In one embodiment of the manual mode, a message may appear after the criteria are satisfied that may direct the user, for example, to “tap when ready,” to alert the user that they may manually initiate picture taking at any time.
Independent of the image and device criteria evaluation, notice may be provided to the user when the foreground image passes one or more document image criteria. For example, alignment notice may be provided to the user when the document image in the foreground of the viewfinder is adequately framed. A technique such as edge or corner detection may be employed for this purpose. The notice may be provided by sound, words, or image, such as an illuminated box. Image capture, however, may occur independent of the notice mechanism or any other document image criterion applied to the foreground image, and may transpire regardless of whether any notice has been provided or other document image criterion applied to the foreground image is satisfied.
After capture, the image of the front of the document may be extracted from the composite image 350. The document image may be shown to the user in the viewfinder 230. The image may be shown in color or as a black and white or grayscale version of the image. If the document image is a check, the user may, by way of example, review the image to ensure that certain check features, such as payee, date, amounts, signature, and MICR line, are clear and legible. If the image is not acceptable to the user, the user may discard it and capture a new image.
The image may be sent automatically to a remote server of for example, a check deposit processing system for processing, or the processing may occur on the device itself. The image 350 may be evaluated relative to a variety of document image criteria, such as height, width, or the presence of edges, corners, or MICR numbers. The processing may also involve electronic reading of the amount of the check, such as through optical character recognition (OCR). If one or more of certain of the document image criteria are not satisfied, the image may be rejected, with a message provided to the user, which may include a suggestion that a new photograph be taken.
Similarly, the amount of the check that is read may be displayed to the user, who may be alerted if that amount and an amount that the user may have entered do not match, and the user may be provided the opportunity to enter a revised amount. If the amount of the check could not be determined, a message may be provided to the user, such as “Deposit Incomplete” or “Amount Required,” indicating that the user must explicitly provide the amount of the check. Typically, processing of the transaction will not be completed if a check amount has not been determined by the software or entered by the user.
The user may also capture an image of the back of the document. This may be preceded by instructions to the user, such as, if the document is a check, to turn over the check to endorse it on the back. Other instructions may be given such as directing that the endorsement be restricted; for example, “for mobile deposit only.” Other static or transient instructions may be given, such as “Sign & Align,” and may include an explicit instruction to capture an image of the back of the document.
As shown in
Capture of the image of the back of the document may proceed as with capture of the front image of the document. The software will monitor the entire, composite image 635 as seen in the viewfinder 230, with the composite image and the device evaluated according to specified criteria. Image and non-image criteria are used. In automatic mode, after a plurality of composite image criteria are satisfied and at least one non-image criterion is satisfied for a specified period or periods of time, an alert may be given to the user, such as “Hold steady,” followed by the device's camera automatically capturing the image, possibly after providing a countdown or similar further alert. If, at any time prior to actual image capture, the composite image or device fails to satisfy any of the criteria, the image capture process may abort without image capture. In manual mode, a message may appear after the image, non-image, and temporal criteria are satisfied that alerts the user that they may now capture an image, and the user may then manually initiate the picture taking at any time while the criteria remain satisfied. The user will be unable to capture an image if any of the required criteria become unmet.
Independent of the criteria evaluation, edge detection may also be separately employed to alert the user when the image of the back of the document 550 in the foreground of the composite image 635 is adequately framed, although image capture need not be dependent upon satisfaction of any such document image criterion.
Once the composite image 635 is captured, the image of the back of the document 550 may be extracted from the composite image and automatically sent to a server such as a server at a check deposit system of a financial institution or deposit processing service, for evaluation, or evaluation may occur on the device. The document image 550 may be evaluated relative to a variety of criteria. For example, the document image may be evaluated for the presence of a signature endorsement. If none appears, or if a restrictive endorsement is required and not detected, a warning may be given to the user or the image may be rejected with instructions to provide the needed endorsement and retake the photo. The image of the back of the document may be evaluated for any of a variety of other criteria, such as edge detection, width or height parameters, and relative width and height to the associated front image, and rejected if one or more criteria are not met, possibly with an instruction to take another photo.
After acceptable images of the front and back of the document are captured, the frontal image is shown to the user. The image of the back of the document may also be shown to the user, but is not necessary. If a check, the detected amount of the check may also be shown.
Prior to submission of the document for processing, the user may be given the option to capture images of one or more additional documents. This may occur, for example, by the tapping of an “Addition” symbol on the device. Capture of those images may proceed as described above.
Upon completion of the image capture process, the user may then provide an instruction for the server to begin processing of the document or documents whose images have been captured. If the document or documents are checks, this may include an instruction to a server related to a check deposit processing system, such as a financial institution or deposit processing service, to submit the images for deposit, such as by tapping a “Submit” symbol or by way of another indication of approval for the deposit to proceed. The device may then provide the user with options, such as including a memo with the deposit. Following submission, the user may receive notice that the processing has been successful, which may include notice that the check or checks have been submitted remotely to a financial institution for deposit. The user may also be provided with information relating to the status, timing, and/or other aspects of the deposit process or other transaction.
Not Applicable.
