Patent Application
20180068196
The present invention relates to optical character recognition. More specifically, the present invention relates to the optical recognition of embossed or raised characters. The present technique can be applied especially to the reading of embossed or raised characters on physical carriers such as for example bank cards.
E-commerce is becoming increasingly unavoidable in daily life. There are several modes of payment in online purchases. These are for example bank cards, intermediary services such as Paypal, or bank transfers. Among known payment modes, bank card payment is the most widespread and is offered by almost all online commercial sites.
When a user purchases goods or items with his bank card on an Internet site, he has to provide information about his bank card. This information can include for example a card number (PAN or Primary Account Number) comprising 16 digits, an expiry date, a name of the bank holder and a verification code (MasterCard CVC or Visa Card CVV). The user must read all this information and enter it by hand onto a web page in order to make his online payment. The visual reading and manual entry of information is complicated and can be affected by human error which would lead to losses of sales volumes.
It has been proposed to apply a classic OCR technique to read the relevant information on the card and enter it automatically on the web page instead of having the user do it. The OCR technique is adapted to recognizing standard texts printed in black ink on flat paper. However, this method fails spectacularly in the reading of information on a bank card. The fact is that the items of information on a bank card are embossed or raised (according to the ISO/IEC 7812 standard) with foil stamping in order to facilitate reading by users. The appearance of the raised letters and digits is therefore very different from that of the letters and digits printed in black ink on flat, white paper. The classic OCR technique, which is adapted to recognizing characters printed on flat, white paper is therefore not capable of recognizing raised characters. Besides, the background of a bank card is often richly colored and decorated. In addition, the ink placed on the characters of the bank card tends to disappear with time and use. It is then very difficult to distinguish one character or another on the card. This raises additional problems for the classic OCR technique.
The “card.io” application proposes a solution that implements a classic OCR technique to recognize raised characters on a bank card using an image captured by a camera. However, because of the problem related to raised characters, this solution calls for specific lighting conditions such as for example lighting through the flash mechanism of a camera or mobile phone. This very soon reduces the charge in the battery of the camera or the mobile telephone. In addition, the application sometimes does not even have reliable results and/or an acceptable response time.
A solution therefore needs to be found, enabling speedy and reliable recognition of raised characters without dictating specific lighting conditions when capturing the image of the raised characters.
The proposed technique does not present the problems of the prior art. More particularly the proposed technique relates to a method of character recognition, a method implemented by means of an electronic device of a type of consisting of the recognition, in an image representing an object comprising at least raised character, called a basic image, of at least one raised character of said basic image.
Such a method comprises:
According to one particular characteristic, said phase for processing said at least one image comprises a step for building a normal map representing normals of the dots on at least one zone of the basic image delivering said identification image.
Unlike a basic image, the normal map no longer comprises information on luminance and colors. The identification of the characters is thus simpler and more reliable.
According to one particular characteristic, said phase for processing said at least one image comprises a step for the generation by the Phong model, preliminarily or on request, of model images of raised characters according to plurality of pre-determined lighting conditions delivering a plurality of sets of reference images;
and said phase for identifying comprises:
According to one particular characteristic, the raised characters are on a bank card in said basic image.
Thus, the characters on the bank card can be recognized by the method without requiring visual reading and manual entry by a user.
According to one particular characteristic, the method comprises a phase of preliminary processing of an image of the bank card captured by camera, delivering said basic image.
According to one particular characteristic, said phase of preliminary processing comprises the steps of:
Thus, the preliminary processing phase improves the quality of the captured image. The characters on the bank card in the basic image are more easily recognized.
According to one particular characteristic, the step for building comprises:
Thus, the lighting condition is determined by means of a flat surface of the bank card. The normal map can be determined simply and reliably.
According to one particular characteristic, the step for determining and the step for computing use the following equation according to the Phong illumination model:
where:
According to one particular characteristic, said step of computation comprises the following steps:
Thus, the ambient component is eliminated. The computation of the normals of the dots of the target zone is simplified.
According to one particular characteristic, said phase for identifying said characters comprises:
The comparison of the vectors is simpler to carry out. It is enough to compute a distance between two vectors. The phase for identifying the characters is thus simplified.
According to one particular characteristic, said known characters comprise capital letters A to Z of the Roman alphabet and the ten Arabic numerals 0 to 9.
Thus, the list of model vectors corresponds only to the frequent characters. It is not necessary to build a large list of model vectors. The step for comparing is thus simplified because the number of model vectors is reduced.
According to one particular characteristic, said current vector and said model vectors have m×n dimensions, each of said normal maps of the known characters having a size of m×n pixels.
According to one particular characteristic, said at least one zone comprises a zone of a card number of said bank card and said list of current characters comprises said card number of said bank card.
According to one particular characteristic, said method furthermore comprises a step for verifying said card number of said bank card according to Luhn's formula.
It is thus possible to verify the validity of the digits of the card number of a determined bank card by the method.
According to another aspect, the invention also relates to an electronic character recognition device, and an electronic device of the type comprising means for the recognition, in an image representing an object comprising at least one raised character, called a basic image, of at least one raised character of said basic image.
Such a device comprises:
According to one specific implementation, the different steps of the methods according to the invention are implemented by one or more software programs or computer programs comprising software instructions to be executed by a processor of a computer device, such as a terminal according to the invention and being designed to command the execution of the different steps of the method.
The invention is therefore also aimed at providing a program capable of being executed by a computer or by a data processor, this program comprising instructions to command the execution of the steps of a method as mentioned here above.
This program can use any programming language whatsoever and can be in the form of source code, object code or intermediate code between source code and object code such as in a partially compiled form or in any other desirable form whatsoever.
The invention is also aimed at providing an information medium readable by a data processor, and comprising instructions of a program as mentioned here above.
The information medium can be any entity or communications terminal whatsoever capable of storing the program. For example, the medium can comprise a storage means such as a ROM, for example, a CD ROM or microelectronic circuit ROM or again a magnetic recording means, for example a floppy disk or a hard disk drive.
Furthermore, the information medium can be a transmissible medium such as an electrical or optical signal that can be conveyed via an electrical or optical cable, by radio or by other means. The program according to the invention can especially be uploaded to an Internet type network.
As an alternative, the information carrier can be an integrated circuit into which the program is incorporated, the circuit being adapted to executing or to being used in the execution of the method in question.
According to one embodiment, the invention is implemented by means of software and/or hardware components. In this regard, the term “module” can correspond in this document equally well to a software component as to a hardware component or to a set of hardware and software component.
A software component corresponds to one or more computer programs, one or more sub-programs of a program or more generally to any element of a program or a piece of software capable of implementing a function or a set of functions according to what is described here below for the module concerned. Such a software component is executed by a data processor of a physical entity (terminal, server, gateway, router etc) and is capable of accessing the hardware resources of this physical entity (memories, recording media, communications buses, input/output electronic boards, user interfaces etc.).
In the same way, a hardware component corresponds to any element of a hardware assembly capable of implementing a function or a set of functions according to what is described here below for the module concerned. It can be a programmable hardware component or a component with an integrated processor for the execution of software, for example, an integrated circuit, smart card, a memory card, an electronic board for the execution of firmware etc.
Each component of the system described here above can of course implement its own software modules.
The different embodiments mentioned here above as well as the characteristics that compose them can be combined with one another to implement the invention. For example, the present technique can equally well be implemented on any type of carrier that comprises raised or embossed characters and is in no way limited to bank cards.
Other features and advantages of the invention shall appear more clearly from the following description of two embodiments given by way of simple illustratory and non-exhaustive examples and from the appended drawings, of which:
As explained here above, character recognition software programs have great difficulty in recognizing raised characters whether it is on bank cards or on other carriers (signposts, signs etc.). In general, the computerized processing of an image for the recognition therein of a shape or of characters raises problems once the lighting conditions are poor. The present technique on the contrary ingeniously makes use of lighting conditions to facilitate the recognition of raised characters (alphabetical characters, numerical characters or again characters in Braille). Here below, we shall strive essentially to describe a process of recognition relative on bank cards but it is clear that the present invention can be applied to many other fields especially the recognition of characters in Braille.
The general principle relies on a method of construction based especially on the Phong reflection model. This method is implemented to enable the recognition of raised characters. More particularly, in a first embodiment, the building method is applied to a basic image comprising characters to be recognized and it enables the building of a derived image (called a normal map) comprising the characters to be recognized in a more easily recognizable form (i.e. the image is no longer in relief). In a second embodiment, a building method is applied to a series of characters as a function of determined lighting conditions in order to produce reference images of the raised characters (synthesis images), the reference images being then used to identify characters to be recognized in a basic image.
The use of one or another of these embodiments (or of certain characteristics of these embodiments) to carry out recognition is made possible by several factors. One of the factors lies in a priori knowledge of the thickness of the raised (or embossed) characters to be recognized. Whether it is for the characters of bank cards or again for characters or words in Braille, the standards lay down a thickness for these characters. For bank cards, this standardized thickness of the embossing makes sure that the card can be inserted into card readers for example: each raised character therefore has a constant and known maximum thickness. Besides, another known factor is the flatness of the surface situated outside the characters (for example the surface of a bank card is flat, as is the surface of a text in Braille). As described here below, these two factors enable the determining of a normal map (or altitude map) which is then used to recognize characters more easily.
Thus, in a first embodiment, a normal map is built. This normal map represents normals of the dots in a target zone comprising raised characters in a basic image. A dot in an image corresponds to one pixel of the image. Each pixel of the normal map is actually a vector giving the information on the elevation and inclination of its surface, i.e. the surface of the texel on to which the normal map will be applied. The characters in the target zone can be identified from the normal map (or altitude map). Indeed, the normal map makes it possible to characterize the shapes of the raised characters on a target zone. The characters can be identified more easily and more speedily from the shapes of the raised characters because a normal map (unlike a basic image) no longer comprises information on luminance and colors. The characters identified by the method according to the proposed technique are thus more reliable. In the prior art solution, the characters are identified from an image (a photograph) on which raised characters are captured (on a bank card for example). The information on the luminance and the colors of the image complicate and slow down the identification process and make the result unreliable.
The basic image can be a photograph of an object (a physical carrier) comprising raised characters. The photograph can be captured by any device (for example a camera, a smartphone, a tablet such as an iPad etc.) comprising a camera. According to another embodiment, this basic image can result from a preliminary phase for processing a captured photograph. This preliminary processing comprises any type of operation to edit the photograph (trimming, rotation, adjusting settings, distortion etc.) and delivers a basic image. The building phase can thus be carried from an image that is more reliable and adapted to optical recognition.
In general, as illustrated with reference to
When the Phong model is used for processing the basic image, as referred to in the first embodiment here above, the processing phase 11 is implemented to process the basic image (P2) and it comprises:
and the identification phase 12 comprises:
When the Phong model is processed for reference images, as mentioned in the second embodiment here above, the processing phase 11 comprises:
and the identification phase 12 comprises:
Here below, a detailed description is provided of the three phases of the method according to the proposed technique. The captured image is considered to be a photograph of a bank card comprising embossed (raised) characters (for example a PAN number, an expiry date and/or a holder's name).
The preliminary processing of the image of the bank card comprises the operations described here below with reference to
The preliminary processing described here above can be performed by using the “OpenCV” library.
It must be noted that the preliminary processing phase is not always necessary. Indeed, when the captured image is of high quality (the bank card is appreciably rectangular), it is possible to directly execute the phase for building the normal map on the basis of this captured image.
5.2.2.1. Phong Illumination Model
The building phase relies on the Phong illumination model. The Phong illumination is a local model, i.e. the computation is done at each dot (pixel) of an image. This empirical model makes a credible computation of the light reflected by a dot under study. To this end, it combines three elements: ambient light, scattered light and specular light. The goal is to compute the light intensity that will be sent out by reflection by the dot studied, illuminated by another source assumed to be a pinpoint source, in a precise direction (that of the observer). To this end, the light is separated into three components:
For each material, characteristic constants are defined:
The terms ia, id and is denote the intensity of the ambient, scattered and specular light. Ia, Id and Is are the values of reflected intensity of a studied dot (pixel) of an image. IP is the total of the intensities reflected from this studied dot (pixel).
The following director vectors are defined: {circumflex over (L)} for light, {circumflex over (N)} for the normal of the studied dot on the surface, {circumflex over (R)} for the direction in which the light will be reflected on a mirror, {circumflex over (V)} for the direction of view of the observer.
{circumflex over (R)} is deduced by the following relationship:
{circumflex over (R)}=2({circumflex over (N)}·{circumflex over (L)}){circumflex over (N)}·{circumflex over (L)} (1)
The total of the intensities reflected from a studied dot is represented by the following equation:
I
P
=I
a
+I
d
+I
s (2)
wherein:
I
a
=i
a
·k
a (3)
I
d
=i
d
·k
d({circumflex over (L)}·{circumflex over (N)}) (4)
I
s
=i
s
k
s({circumflex over (R)}·{circumflex over (V)})a (5)
For several light sources, the following complete formula is obtained:
I
P
=i
a
k
a+Σmε{sources}(id,mkd({circumflex over (L)}m·{circumflex over (N)})+is,mks({circumflex over (R)}m·{circumflex over (V)})a) (5)
For the sake of simplicity, it is assumed that there is only one light source. Thus, a simplified equation of the total of the reflected intensities of a studied dot is obtained:
I
P
=i
a
k
a
+i
d
k
d({circumflex over (L)}·{circumflex over (N)})+isks({circumflex over (R)}·{circumflex over (V)})a (6)
By using equation (5) or (6), it is possible to predict the appearance (the reflected intensities of the points in the form of an image) of the raised characters according to a given condition of light (the shape and the material of the raised characters are known beforehand).
This image can be computed in a various ways: this image can be computed for example by using existing libraries such as “OpenGL”.
5.2.2.2. Phase for Building a Normal Map
Here below, referring to
The general principle of this phase for building consists in computing the normal {circumflex over (N)} of a studied dot by using the equation (5) or (6). Indeed, once an image of raised characters on a bank card is captured, the value IP of each dot (pixel) of the image is known. The characteristic constants ka, kd, ks and a of the material of the bank card is also known (because the material of the card is known). The lighting conditions can be deduced by using the value IP of a dot on a flat surface of the image of the bank card.
All the dots on the flat surface SP have a known normal R that is perpendicular to the surface SP. The lighting conditions (including especially the director vector {circumflex over (L)}) can also be determined 111 by implementing the Phong illumination model (equation (5) or (6) on a dot on the flat surface SP (the normal {circumflex over (N)} of the point is known).
When a dot is analyzed in the zone of the raised characters CR, the ambient component iaka can be eliminated by subtracting the minimum value of light intensity of the dots on the bank card. If we take the equation (6) at the outset, we thus obtain:
I
P
−i
a
k
a
=i
d
k
d({circumflex over (L)}·{circumflex over (N)})+isks({circumflex over (R)}·{circumflex over (V)})a) (7)
The part on the right-hand side of the equation comprises the scattered component (idkd ({circumflex over (L)}·{circumflex over (N)})) and the specular component (isks({circumflex over (R)}·{circumflex over (V)})a). These two components depend solely on the normal {circumflex over (N)} (the other parameters are known). The normal {circumflex over (N)}, which corresponds best to the light intensity of each dot in the zone of the raised characters of the bank card can be computed 112 by using standard minimizing techniques. The director vector of the normal {circumflex over (N)} of each dot in the zone of the raised characters can be recorded in a normal map P3 (or altitude map).
According to another embodiment, the normal map (or altitude map) can cover the entire surface of the bank card (not only the zone of the raised characters). This normal map (or altitude map) characterizes the shape of the surface of the bank card comprising the raised characters. The characters can thus be identified from this normal map P3 (or altitude map) in the identification phase described in the section below.
As explained here above, the normal map (or altitude map) characterizes the shape of the raised characters on the bank card. It is thus possible to recognize the use of the known OCR algorithms for identifying characters from a normal map (or altitude map).
According to one embodiment of the proposed technique, the character identifying phase is carried out by using an algorithm that requires fewer computation resources. According to this algorithm, the phase for identifying characters comprises the following steps (illustrated with reference to
The characters identified can thereafter be used for the automatic filling of the entry zones of a form displayed on a screen; this method of action enables the user to verify that the phase of recognition has taken place properly. The recognized characters can be used in the various applications such as for example online payment. In this case, other operations can be conducted subsequently to the recognition of characters in order to make sure that the recognized data is accurate.
Thus, for example, the raised characters displayed on a bank card normally comprise a number of the card (e.g. PAN (Primary Account Number)) the apparent validity of which can be verified by Luhn's formula which is also known as the “modulo 10” algorithm. This is a checksum formula used to validate bank card numbers. Thus, the phase for identifying the characters of the bank card number can include a step for verifying the number of the bank card by means of this “modulo 10” algorithm. If the detected number is not valid, a new step of comparison can be executed. In this new step of comparison, a model vector is identified for each new current vector. This model vector is for example the second closest to the current vector (the model vector that is at the second smallest distance from the current vector). Among the model vectors identified, the one at the smallest distance from its corresponding current vector is selected. The digit represented by the selected model vector replaces the corresponding digit in the non-valid number of the bank card.
According to another embodiment of the proposed technique, in the step for comparing, each current vector is compared with all the vectors in the list of model vectors. The first two or three model vectors which are the closest to the current vector (having the smallest distances from the current vector) can be selected. Thus we obtain two or three model vectors for each current vector. In other words, each current vector can probably represent one of the two or three digits represented by the two or three model vectors. In the determining step, the digits of a bank card number are determined according to two criteria:
The reliability of the method is thus improved.
The preceding sections describe the building of a normal map and the identification of the raised characters from the normal map.
In another embodiment of the proposed technique, it is not necessary to rebuild the normal map. The raised characters in the basic image can be identified according to a specific method for recognizing raised characters. This embodiment uses models of characters that have been pre-illuminated, according to pre-determined lighting conditions (for example about forty lighting conditions). Thus, for each character, there is a series of model images of this character, each model image corresponding to a particular condition of lighting. These model images are called reference images ((IRxx).
This method of recognition comprises the following steps:
According to one particular embodiment, the method, subsequently to the step for obtaining, can comprise a step for determining a lighting condition of the basic image of the bank card. One example of determining a lighting condition by means of a flat surface of the bank card is described in detail further above.
Once the condition of the lighting of the basic image is determined, the method comprises a step for filtering sets of reference images in order to select, among the sets of existing reference images, only a reduced number of sets corresponding to the lighting conditions of the basic image. Ideally, only one set of reference images is selected, constituting all the identification images. In this case, a character recognized on the bank card corresponds to a single identification image of the set of identification images: the list of identification images is thus constituted by a single set of reference images. The step for determining the characters is made easier because the number of identification images is greatly reduced (there is only one set of identification images).
As a complement, Luhn's formula can also be used to verify the validity of the bank card number. Indeed, the raised characters on the bank card can necessarily include the digits of a bank card number.
It can also be noted that the (synthesis) reference images can be generated on the fly if the apparatus (on which the recognition is implemented) possesses sufficient computation capacities. One promising feature of this variant is that it offers a compromise between storage capacities (there is no longer need to store N models in advance, since N is potentially very great) and computation capacities (henceforth only the images that are necessary have to be computed). The other promising feature of this variant is that it is possible to determine the lighting conditions more finely by an optimization method such as the expectation-maximization method. Concretely: the algorithm minimizes, one after the other, the lighting error (measured by comparison with the non-raised zone of the card) and the error in the reading of the characters (measured by comparing the reference image and the raised zone, possibly with penalties if the Luhn code is incorrect).
Referring to
The device comprises a memory M comprising a buffer memory, a processing unit P equipped for example with a processor and driven by an application or a computer program Mp implementing the method of recognizing raised characters according to one embodiment of the proposed technique.
At initialization, the code instructions of the computer program Mp are for example loaded into a RAM and then executed by the processor of the processing unit P. The processing unit P inputs a signal sent out by a transmitter. The microprocessor of the processing unit P implements the methods of recognition of raised characters according to the instructions of the computer program Mp.
Preferably, the device comprises a camera used to capture an image or the object (for example a bank card) comprising raised characters. The processing unit of the device can subsequently execute the instructions of the computers to implement the character recognition method. The device (with or without camera) can include a wire communications module (USB, Ethernet, etc.) or wireless communications module (Wifi, Bluetooth®, etc.) to receive an image of a bank card coming from another electronic device (a camera, a computer, a server etc.).
The examples of embodiments described here above are obviously given only purely by way of an indication. Those skilled in the art are capable of understanding the advantages procured by the present technique and especially the advantages procured by the method for recognizing raised characters. It is clear that such a method can be used in another context without departing from the scope of the present technique.
| Number | Date | Country | Kind |
|---|---|---|---|
| 1658228 | Sep 2016 | FR | national |
