The present invention relates to a method for digital capture of information present on a medium, by means of a camera fitting out a communications terminal. Its object is to enable the terminal to store and/or transmit this information to an addressee, it being understood that in order for it to be able to be used, this information should be extracted and corrected to notably take projective distortions into account and/or completed by incorporating a background and/or textual data.
It is notably but not exclusively applied to transmitting and storing textual data and digitally scanned graphics, as previewed by a camera under any incidence and then processed with correction of the projective distortion and possibly with enhancement of the resolution.
Such a process is most particularly suitable for transmitting textual and/or graphic information taken by a camera fitting out a portable communications terminal such as a cellular radio transmitter/receiver for example.
Indeed, for this type of applications, when one desires to transmit a written message extracted from a photograph to an addressee, it is frequently required that corrections be made without which the message appearing in the photograph received by the addressee, would be illegible. Also, it turns out to be desirable to proceed with inclusions of external patterns, for example captions inputted on the keyboard of the device, into the transmitted image, or even to perform superimposition of the image taken by the camera and of a background which may for example be selected in a library accessible by the unit.
It is generally known that information has become ubiquitous today and that its control is essential; now this information largely consists of text data.
Knowledge, whether it be technical, scientific, historical, economic, legal, medical knowledge, is mostly stored and conveyed by texts; recently published knowledge is directly accessible in electronic form; on the other hand, the majority of inherited knowledge is still only available in paper document form.
Society is thus confronted with an enormous reprocessing need, also called retroconversion, for changing over to an electronic format.
Document recognition is related to image recognition; it concerns all matters about written language and its digital transformation: character recognition, text formatting, content structuring and accessing information through its indexation.
So it is a matter of rediscovering an existing structure so that the recognition is guided by an explicit or implicit model of the investigated document class. The model describes the items which compose the document and their relationships, this description may be physical such as by giving the page make-up format.
Moreover, it is known that interpretation by a person of a text or a graphic previewed by a camera, assumes a quasi-normal or perpendicular shot relative to the document bearing the text or graphic and sufficient resolution for distinguishing details.
It may easily be understood that the reading of a text by the person receiving the message is largely facilitated under normal or quasi-normal incidence relatively to the plane of the document; as for the interpretation of a graphic, it almost inevitably requires compliance with shapes and proportions.
Finally, character and document recognition has made considerable progress; the scanners provide sufficient resolution for subsequent recognition steps; the latter are the following:
- acquisition or digitization,
- straightening-up,
- quantification,
- binarization,
- page segmentation,
- character recognition,
- logical structure recognition.
To enhance the quality of segmentation and automatic character recognition, it is desirable that the image of the document be perfectly straight and with sufficient resolution; this notably facilitates searching for columns of the text, when two consecutive columns are very close to each other, and recognizing characters if the latter are of a particularly reduced size; the global offset angle of the page therefore needs to be detected and the definition of the image needs to be enhanced, notably, of those coming from a camera with insufficient quality for distinguishing the details of a text or a graphic taken at a certain distance and for guaranteeing a minimum resolution for recognizing the characters; several algorithms have been developed for detecting the tilt angle of the text; however, the latter should not exceed 10-20° in the scanning plane.
The difficulty becomes insurmountable when the document has been viewed by a camera under any incidence, as the document has undergone a projective distortion: starting from a certain distance of the camera, it is seen that details in the image which are required for recognizing characters and consequently for understanding the document, disappear.
More specifically, the object of the invention is to abolish these drawbacks and to allow corrected information, possibly completed by inclusion of a background and/or textual data, to be stored and/or transmitted to an addressee.
Of course, to achieve this result, the invention proposes a solution taking into account constraints due to the size of a standard communication terminal, to both hardware and software resources and transmission rates.
Accordingly, the method according to the invention comprises the following steps:
- taking at least one image by the camera,
- at least partially extracting identifiable contextual data included in said image by processing means integrated into said terminal,
- extracting raw data relating to said information by said processing means,
- storing raw data in a memory of said terminal and/or transmitting them to a receiver,
- correcting raw data by processing means of said terminal and/or said receiver with the help of contextual data,
- transmitting corrected data to the addressee by said terminal or by said receiver.
Advantageously:
- this method may comprise the taking of several images and the merging or selection of extracted data, before or after correction,
- contextual data and raw data may be transferred to the aforementioned receiver, which may make the aforementioned corrections and transmit the corrected data to the addressee, as requested from the aforementioned terminal,
- corrections may be made by the processing means of the terminal while corrected data may be transmitted to the addressee directly by the terminal or indirectly via the receiver,
- contextual data may be transmitted to the receiver, which may perform processing of these data and transmit control instructions to the terminal allowing the processing means of the terminal to make corrections to the raw data,
- contextual data and raw data relative to said information may be transmitted to the receiver, which may make the aforementioned correction and transmit the corrected data to the addressee as well as control instructions allowing the processing means of the terminal to themselves make the corrections to the raw data,
- the aforementioned terminal may comprise means for accessing a bank of images as well as means for carrying out keyed insertion of the corrected data into at least one selected image.
Moreover, the correction step provided in the method according to the invention, may comprise the following operative phases:
- producing a contour in the document to be processed or around the document to be processed, either by a manually produced or printed plot (for example, a quadrilateral, a rectangle) or with the help of any recessed material frame,
- presenting the document to be processed in front of the camera under any incidence so that the aforementioned contour is entirely visible in the image present on the viewing screen,
- contour searching in the image,
- computing projective distortions, extracting and merging contents of the images,
- generating the final image.
Advantageously, with this method, it is possible to:
- facilitate interpretation of the received document by the relevant person,
- rebuild the structure of the document from the physical description of the latter,
- perform character recognition from software packages known from the state of the art,
- transfer the document via a communications network, such as Internet, a cellular network, such as a GSM, GPRS or UMTS network,
- store the document on a suitable medium as known from the state of the art,
- reduce the size of the digitized information so as to reduce the memory required for its storing and increase the transmission rate of this information.
In this case, the method according to the invention may involve:
- a central unit grouping together processing and storage means,
- a camera connected to the central unit, preferably attached or integrated to the latter,
- a screen for viewing the image taken by a camera,
- means for transmitting and storing the digitized information.
The retroconversion of the document, i.e., its conversion to an electronic format, will thereby be made possible, therefore allowing it to be utilized, transmitted and stored.
Embodiments of the invention will be described hereafter, as non-limiting examples, with reference to the appended drawings, wherein:
FIG. 1 is a schematic representation of a system for extracting and correcting information contained in an image taken by a communications terminal fitted out with a camera;
FIG. 2 is a schematic representation with which the problems posed by making shots of a document under any incidence may be illustrated;
FIG. 3 represents a flow chart relating to the acquisition of the image and to the search for the contour in the image;
FIG. 4 represents a flow chart relating to the extraction, merging of the contents of the images, and generation of the final image;
FIG. 5 represents a detailed flow chart relating to the contour search in the image;
FIG. 6 represents a detailed flow chart relating to the selection of the contour and the computation of the projective distortion of the contour found in the image;
FIG. 7 represents a detailed flow chart relating to the merging of information contained in the found contour and the enhancement of contrasts of the images;
FIG. 8 represents a detailed flow chart relating to obtaining the final image;
FIG. 9 is a schematic representation illustrating a mode for selecting the contour as a graphic;
FIG. 10 is a schematic representation illustrating another mode for selecting the contour.
In the example represented in FIG. 1, the system for applying the method according to the invention involves a communications terminal TC, including a transmitter TR such as for example, a GSM mobile phone, conventionally comprising an emitter E1 and a receiver R1. This TC terminal is fitted out with a digital camera CN for making shots of a medium O comprising textual data DT and contextual data CD.
According to the invention, digital data delivered by the CN camera, for each of the images of medium O, are transmitted to a processing circuit comprising a device for extracting contextual data EC (which may consist of a contour inscribed in medium O, for example, a document which one desires to process) and a device for extracting raw textual data EDTB relative to the information contained in the image. This extraction device EDTB is designed so that it may possibly use the contextual data extracted by the extraction device EC.
The extraction device EDTB is connected to a correcting circuit CC which is designed so as to at least partially correct the raw data delivered by the extraction device EDTB from the contextual data delivered by the extraction device EC.
The data corrected by the correcting circuit CC are transmitted to the emitter E1 of transmitter TR in order to be retransmitted to an addressee DES, either directly, or via the receiving device REC located at a distance from transmitter TR.
The receiving device REC is equipped with a processing circuit TRC for correcting raw data, possibly partly corrected by the correcting circuit CC of the communication terminal TC. This correction is made with the help of contextual data extracted by the extraction device EC and transmitted to the receiving device REC by terminal TC. Also, this receiving device REC may be fitted out with a system for automatic writing recognition in order to be able to reuse the information in a text editor.
Alternatively, the receiving device REC may be designed in order to develop processing instructions or algorithms from the contextual data transmitted by terminal TC and for transmitting these instructions or these algorithms to the correcting circuit CC, via an emitter E2 and receiver R1, so as to allow the TC terminal to make the corrections to the raw data by means of the simplified correcting circuit CC (unwieldy processing operations which require significant resources being performed by the processing circuit TRC of the receiving device REC).
The data corrected by the correcting circuit CC or by the TRC processing circuit, may be transmitted to a keyed insertion circuit CI located upstream from the transmitter TR which enables these corrected data to be included or possibly merged into at least an image selected by the SEL selection circuit. Conversely, the keyed insertion circuit may comprise means for incorporating into said selected image, other pieces of information such as for example textual and/or graphic information.
This image may for example consist of a monochrome background. It may be selected from a plurality of images stored or possibly downloaded in terminal TC, or even from those taken by the camera.
Advantageously, these images may be derived from an image data bank BDI fitting out receiver REC.
In this case, the TC terminal may be designed so as to send to the REC receiver, a command for selecting an image contained in the BDI data bank.
Also, the REC receiver will be designed in order to send to the TC terminal, the selected image for performing the keyed insertion.
Of course, terminal TC may comprise a display AF for viewing the data and possibly the contextual data, possibly inserted in an image, before their being transmitted to the DES addressee, either directly, or via the REC receiver.
In the example illustrated in FIG. 2, camera C is centered on the center of a rectangular document D according to an incidence angle i. The image of this document, captured by camera C as viewed on screen E, has undergone a projective distortion and therefore has a trapezoidal shape D′.
To abolish this drawback, the invention proposes prior inclusion of contextual data into the document D, here a closed contour in the document to be processed or around the document to be processed. This contour may also consist of the peripheral edge of the document; it thereby forms a digitization zone viewed by the camera; the successive images acquired by the camera are made visible on the viewing screen.
According to the flow chart of FIG. 3, for each of the shots requested by the user, the method comprises a first search step for the contour (block 1), until the found contour is obtained (block 2); the contour having been detected, the image is saved (block 3) and acquisition is completed; the process is repeated until N images are obtained, N being set beforehand (block 4).
According to the flow chart of FIG. 4, from saved images, the method performs a computation of the projective distortions of the contours (block 5), and then the merging of the information contained in the images (block 6) and finally the generation of the final image (block 7).
According to the flow chart of FIG. 5, the method comprises the detection of boundaries present in the image (block 11), the extraction of sufficiently long boundaries (block 12) and the detection of zones delimited by the found boundaries with sufficient surface area and not touching the edge of the image (block 13).
According to the flow chart of FIG. 6, the method proposes for each zone found in the contour search phase, computing the main axis of the zone (block 51), then finding an external point to the zone on the axis (block 52), then building the external cone from the external point (block 53), then extracting the points from the boundary, the external normal of which is opposed to the vector which joins it and starts from the external point (block 54), then computing the line borne by the main axis of the extracted points (block 55), and then after finding four lines, computing the four apices of the quadrilateral from the four lines (block 56), and then, as the quadrilateral surface area is close to the surface area of the zone, computing the homography distorting the quadrilateral into a rectangle with a proportion set beforehand (block 57).
If four lines are not found, the method proposes computing the axis inferred by translation of the main axis of the extracted points, in the direction perpendicular to the latter (block 58).
If the surface area of the rectangular quadrilateral is not close to the surface area of the yet unconsidered zone, the method proposes again performing a computation of the main axis of the zone (block 51), followed by the operations as defined above.
According to the flow chart of FIG. 7, the method for each image to be processed by correcting the projective distortion, comprises the building of a rectangular virtual image by projecting the contents of the contour by using the computed homography (block 61), the enhancement of the contrast of the virtual image by applying a so-called edge enhancement filter (block 62), and then computing the average virtual image for which the color intensities are the averages of the color intensities of the enhanced virtual images (block 63).
According to the flow chart of FIG. 8, the method proposes for each pixel of the average virtual image, computing the average of the color. intensities, according to the formula M=(R+G+B)/3 (block 71). If the term M is less than a predetermined threshold, the pixel of the final image is considered to be black (block 72); on the contrary, if the term M is larger than this same threshold, the pixel of the final image is considered to be white (block 73).
The sought-after contour 1 is illustrated in FIG. 9, by four line segments 1a, 1b, 1c, and 1d;
- i.e. line 2, the main axis of the zone delimited by contour 1, passing through the center of gravity G,
- i.e. point 3, located outside the zone on axis 2,
- i.e. half-lines 31 and 32, from point 3, forming the cone external to the line segment 1a of contour 1,
- i.e. points 4 and 5, located on segment 1a,
- i.e. the external normals to contour 41 and 51, the half-lines perpendicular to the line segment 1a, at points 4 and 5.
Points 4 and 5 are two points among the extracted points which define the boundary of the zone materialized by the line segment 1a, as the normals external to contours 41 and 51 are opposed to half-lines joining points 3 and 4, 3 and 5, respectively.
The search for other line segments 1b, 1c, and 1d, is performed according to the same method from line 6, inferred by translation of the line segment 1a, by moving away from the latter relatively to point 3, and from point 7, located on line 6, outside the zone delimited by contour 1, and half-lines 71 and 72 forming the cone external to the line segment 1b.
In this example, the method for selecting a camera digitizing zone for correcting the projective distortion, for enhancing the resolution, then for binarization comprises:
- achieving a closed contour in the document to be processed or around the document to be processed, either by a manually produced or printed plot (for example, a quadrilateral, a rectangle), or with the help of any recessed material frame,
- presenting the document to be processed in front of the camera under any incidence so that the aforementioned contour and the aforementioned document are entirely visible in the image present on the viewing screen,
- detecting boundaries present in the image,
- extracting sufficiently long boundaries,
- detecting zones delimited by the found boundaries with a sufficient surface area and not touching the edge of the image,
- searching for new boundaries and proceeding with the process if the contour is not found, until obtaining a contour allowing the image to be saved and acquired,
- saving and acquiring the image if a contour is found,
- computing the projective distortions of the contours consisting of computing the main axis of the zone, then finding a point external to the zone on the axis, then building the external cone from the external point, then extracting the point of the boundary for which the external normal is opposed to the vector which joins it and starts from the external point, then computing the line borne by the main axis of the extracted point, and then, four lines having been found, computing the four apices of the quadrilateral derived from the four lines, then as the surface area of the rectangular quadrilateral is close to the surface area of the yet unconsidered zone, computing the homography distorting the quadrilateral into a rectangle with a proportion set beforehand,
- computing the axis inferred by translation of the main axis of the extracted point, in the direction perpendicular to the latter, if the four lines are not found,
- again computing the main axis of a yet unconsidered zone, followed by the previous operation, if the surface area of the rectangular quadrilateral is not close to the surface area of the zone,
- building for each image a rectangular virtual image by projecting the contents of the contour by using the computed homography,
- enhancing the contrast of the virtual image by applying a so-called edge enhancement filter, computing the average virtual image for which color intensities are the averages of color intensities of the enhanced virtual images,
- computing the average virtual image for which color intensities are the averages of color intensities of the enhanced virtual images,
- computing, for each pixel of the average virtual image, the average M of the color intensities,
- designating as black pixel, any pixel for which the term M is less than a predetermined threshold,
- designating as white pixel, any pixel for which the term M is larger than a predetermined threshold.
Hence, this method provides presentation, utilization, transmission and storage of texts and digitized graphics, previewed by a camera under any incidence, then processed by correcting the projective distortion and enhancing the resolution.
Of course, the invention is not limited to the above described embodiment.
Thus, notably, the contextual data may comprise an unclosed contour drawn by hand. In this case, detection of this contour CO may be carried out according to an operating sequence comprising the following steps (FIG. 10):
- searching along an horizontal line, for example the center line of the image of a pixel having with the surrounding pixel(s), for a significant change in of level (for example, as determined by the first derivative of the grey levels of the image at these pixels),
- if no pixel is found in the previous step on the horizontal line, similar searching along at least one vertical line (for example, the center line),
- tracking the assumed curve formed by the pixels with significant level variation, by iteratively testing the still unexplored neighboring pixels,
- computing the center of gravity G and the main axes XX′-YY′ of the previously determined contour,
- testing for determining whether the points of the contour have a reasonable dispersion ratio between both main directions,
- selecting an axis of the contour (this axis will preferably be a main axis XX′-YY′ of the contour, and if necessary, a secondary axis) and, on this axis:
- determination on either side of the contour CO of the external points PE1 and PE2 located outside the contour CO at a distance of a few pixels and, for each external point,
- determination of both pairs of singular points (PS1, PS′1)-(PS2, PS′2) such as for example, angles or ends of the contour, by using for this purpose, for each of the external points PE1, PE2, two line segments (SD1, SD′1)-(SD2, SD′2) from an external point and defining an angle in which the contour is inscribed, and
- computing the projective transformation parameters by using the geometrical shape defined by the singular points (PS1, PS′1)-(PS2, PS′2).
In the example of FIG. 10, the contour CO approximately has the shape of a laid down U. In this case, the singular points consist of both ends PS′1, PS′2 of the contour CO and of both apices PS1, PS2 of the angles formed between the core and both legs of the U. The main axis XX′ is not used because it only intersects the contour once. This is why the YY′ axis is used (which again intersects the main axis XX′ at the center of gravity G).
Moreover, the method according to the invention may comprise a process for classifying the medium of the image (plain paper/square-ruled paper) and for removing the squaring in the case of square-ruled paper.
This process consists of determining whether the plots of the low gradient (grey level variation) image form squaring extending to at least a boundary of the image. If this is the case, the method consists of raising the threshold beyond which the gradients are taken into account in order to remove the squaring. Of course, this process implies that the squaring lines have a lower contrast (with reference to the paper) than the hand-written contents of the image, which is true in the very large majority of the cases.
This process may comprise the following steps:
- selecting the smallest significant gradient threshold as regards noise for detecting the contour for example in the way indicated above,
- if the detected patterns touch a boundary surrounding the image, inferring that the squaring lines are present on the medium,
- if squaring lines are present, incrementing the gradient threshold and then again performing the second step with the new threshold, this process being repeated until the found contour no longer touches the edge,
- using the last gradient threshold (the gradient for having the squaring
- lines disappear) in order to extract the data contained in the image (for example according to a standard extraction process) without taking the squaring lines into consideration.
Also, the process for extracting data may be carried out according to a sequence comprising the following steps:
- a) determining, for each point of the image, a value V0[C,L] consisting in a combination of color components of the image for the point located by column C and line L of the image, this value V0[C,L] being expressed as:
V0[C,L]=αRed[C,L]+βGreen[C,L]+γBlue[C,L]
A formula wherein α, β, γ are coefficients which may for example satisfy the following relationships:
α+β+γ=1 and α,β,γ≧0
- b) computing, for each point of the image, a value VN+1 [C,L] in the following way (according to whether this is dark information on a bright background or vice versa):
- c) iterating step b, a predetermined number of times, then taking the final value VNfinal into account
- d) computing the difference D[C,L] for each point of the image
D[C,L]=VNfinal[C,L]−V0[C,L] (or V0[C,L]−VNfinal[C,L]) - e) comparing for each point of the image, the value D[C,L] with a threshold value VS so as to determine the values to be extracted, in the following way:
- if D[C,L]<VS, then D[C,L]=0
- if D[C,L]≧VS, the value D[C,L] is retained or replaced with D[C,L]−VS
- f) the values of D[C,L] are quantified in a predetermined number of levels (it being understood that binarization is achieved if the number of levels is equal to 2).
The threshold value (VS) may possibly consist of the gradient threshold value for disappearance of the squaring, used in the squaring removal process described above.
Patent Application
20060164517
