Patent Application
20220392240
Present invention has its application in the telecommunication sector, within the field of digital information security and digital content processing and generally relates to digital watermarking technology.
More particularly, the present invention refers to a method for detecting, using artificial intelligence (AI) algorithms, document regions that will be excluded from an embedding process. The method enables detecting invalid text contents to be moved by a digital text watermarking algorithm in order to improve the imperceptibility of the embedded information.
The inclusion of hidden information in a text file is a desired feature in many fields and for many different applications. On one side, hidden information embedded in a text document may be decoded by machines to enable automatic classification of documents. At the same time, such information may enable automatic validation of the authenticity, integrity, and ownership of a document. Furthermore, that information—included in confidential documents—may allow their tracking, dissuade information leak, or help in the identification of the responsible in a case of leak.
Information included in documents with this purpose is usually intended to be imperceptible by human beings while appreciable by computer systems, which can read and decode them. Such information inclusion is usually referred to as a watermark. A watermark is an alteration of a document that may include an identifying image or pattern, such as character spacing or character deformation in the case of text, or pixel shifting in frequency or space in the case of images. These alterations are intended to be not noticeable, but until now, any method proposed does not solve the problem of minimizing the alteration effects for a wide range of different document contents (for instance, documents with tables, lists, different indentations, math equations, index, etc.).
In the field of digital watermarking for images and video are well known, for instance, methods based on frequency that do that in an indirect way, where the information is embedded in the high frequency components. In these high frequency components, the visual human system (VHS) is not able to detect smooth modifications, then the perceptibility of the embedded information is very low.
On the other hand, all the digital watermarking algorithms for text documents in the state of the art and patents, specifically those based on shift of the elements such as characters, words, or lines, do not show any mechanism to deal with those kinds of problems, and all methods make the assumption that the documents have been made with only regular text, uniform spacing between elements and other properties that do not exist in the real world. When these methods are used on real documents there exist high probabilities of achieving undesirable visual effects on the documents used in the embedded process (mainly misalignments in elements such as indexes, tables, and lists, or even overlapping between elements such as characters or words). This limitation is the main reason why digital text watermarking algorithms have not made their leap to the market as they have done with other types of multimedia formats, such as images or videos.
Furthermore, the wrong selection of the elements to be used in the embedding process not only affects the visual part, but also can affect the capacity of the system to extract the information. The main reason is that algorithms are extracting information from parts where the spacing (information) is not as expected, so that these unexpected spaces act as noise on the rest of the extracted information, reducing the chances of extracting information.
A third effect to take into account is the amount of information that the system believes can be embedded in the document. The amount of information that can be embedded is related to the amount of text in the document (pages to be more exact). If the algorithms count text that is not valid to hide information on it, the amount of information that can be extracted is less than the algorithm expects, and the results when extracting the information are likely to be unexpected.
Therefore, it is highly desirable to develop a method for detecting regions that will be excluded from the embedding process of digital text watermarking algorithms for above reasons: 1) reduce unwanted visual artifacts in embedded documents, 2) increase the capacity of the system to extract the information from the previous watermarked documents and 3) achieve more accurate data about how much information can be inserted and extracted in a specific content taking into account all the diversity offered by the contents of text documents.
There are known some patents and patent applications in this field.
US2020042785A1 discloses methods, systems, and computer program products for table recognition in PDF documents. A computer-implemented method includes discretizing one or more contiguous areas of a PDF document; identifying one or more white-space separator lines within the one or more discretized contiguous areas of the PDF document; detecting one or more candidate table regions within the one or more discretized contiguous areas of the PDF document by clustering the one or more white-space separator lines into one or more grids; and outputting at least one of the candidate table regions as a finalized table in accordance with scores assigned to each of the one or more candidate table regions based on (i) border information and (ii) cell structure information.
EP2807608A1 provides a borderless table detection engine and associated method for identifying borderless tables appearing in data extracted from a fixed format document. Due to the lack of visible borders, reliable automated detection of a borderless table is difficult. The borderless table detection engine uses whitespace, rather than content, to detect borderless table candidates. Applying heuristic analysis, the borderless table detection engine discards borderless table candidates with a layout that lacks sufficient characteristics of a table and is unlikely to be a valid borderless table.
These two prior art solutions only identify tables in the input PDF files, and no other text elements, such as bullet lists, justified text, text in images, etc. Furthermore, they make use of heuristics for the table identification, and not of AI elements, which has been proven to be much more efficient in the solution of this kind of problems.
U.S. Ser. No. 10/817,717B2 relates to a method and a device for parsing a table in a document image. The method comprises the following steps: inputting a document image to be parsed which includes one or more table areas into the electronic device; detecting, by the electronic device, a table area in the document image by using a pre-trained table detection model; detecting, by the electronic device, internal text blocks included in the table area by using a pre-trained text detection model; determining, by the electronic device, a space structure of the table; and performing text recognition on a text block in each cell according to the space structure of the table, so as to obtain editable structured data by parsing. The method and the device of the present application can be applied to various tables such as line-including tables or line-excluding tables or black-and-white tables. This solution only identifies tables in the input document files, and no other text elements, such as bullet lists, justified text, text in images, etc.
U.S. Pat. No. 9,697,423B1 provides a method for image processing that includes obtaining an image including a table; identifying a first plurality of geometric lines in the image; grouping the first plurality of geometric lines into a plurality of clusters; determining a plurality of hand-drawn lines in the image corresponding to the table from the plurality of clusters; calculating a plurality of points for the plurality of hand-drawn lines; and determining a geometry of the table based on the plurality of points. As the solutions indicated before, this method only identifies tables in the input PDF files, and no other text elements, such as bullet lists, justified text, text in images, etc. Furthermore, it makes use of heuristics for the table identification, and not of AI elements.
WO2017160654A3 discloses a method for extracting data from PDF files. The method includes identifying at least one document identifier associated with a first document in a PDF file. The method further includes determining, using the at least one document identifier, a reference point identifier for identifying a reference point in the first document, an offset value for indicating a location of a first detection area in the first document, and size information for indicating a size of the first detection area in the first document. The method also includes identifying, using a reference point identifier, the reference point in the first document. The method further includes identifying, using the offset value and the size information, the first detection area in the first document and extracting, by processing binary data of the PDF file, data within the first detection area of the first document.
Other solutions or strategies are known by U.S. Pat. No. 9,348,848B2, U.S. Pat. No. 9,268,999B2, U.S. Pat. No. 8,645,819B2, US2008084573A1, U.S. Ser. No. 10/445,615B2 and U.S. Ser. No. 10/592,738B2.
More efficient and faster methods, based on AI or computer vision techniques, able to deal with geometric distortions on a page of a document, with documents including text or not, and/or with vectorial PDFs, are therefore needed.
To that end the present invention proposes, according to one aspect, a method for detecting document regions that will be excluded from an embedding process. The method comprises converting, by an adapter module, at least one page of a received document into a visual representation thereof, the visual representation keeping the position of the characters of the at least one page; receiving, by a text detector, the visual representation; processing, by the text detector, the visual representation using one or more artificial intelligence algorithms, and returning a list of invalid regions with their associated page positions as a result; and using, by a watermark embedding module or by a watermark extracting module, the list of invalid regions to provide a watermarked document or a message embedded in the document.
According to the invention, the document can comprise a digital or a digitalized document.
In an embodiment, each invalid region of the list of invalid regions has associated thereto a confidence score.
In an embodiment, the visual representation comprises an image.
In an embodiment, the image is a binary image where pixels associated to the characters are represented using a first color and where pixels associated to background are represented using a second color. In some embodiments, the first color can be white, and the second color can be black.
In an embodiment, the at least one page is in image format, wherein the converting step further comprises correcting a skew of the page by means of using a recognition algorithm. For instance, the recognition algorithm can be the Hough Transform.
In an embodiment, before performing the correcting step, the method comprises preprocessing the at least one page by means of applying a Canny Edge Detection algorithm and different morphological operations on the at least one page. Moreover, a resizing of the at least one page to a lower resolution can be made.
In an embodiment, the position of the characters can be obtained by means of using an image-based segmentation algorithm, for example a projection-based algorithm.
In an embodiment, the artificial intelligence algorithms comprise a deep neural network, and optionally a supervised learning algorithm. The deep neural network can be based either on a detection-based architecture or on a segmentation-based architecture.
In an embodiment, the list of invalid regions comprises a set of boxes where each box is defined by: an ‘x’ value indicating a left-up corner of the box in a horizontal coordinate, a ‘y’ value indicating a left-up corner of the box in a vertical coordinate, a ‘w’ value indicating a width of the box in pixels, and an ‘h’ value indicating a height of the box in pixels.
Other embodiments of the invention that are disclosed herein also include software programs to perform the method embodiment steps and operations summarized above and disclosed in detail below. More particularly, a computer program product is one embodiment that has a computer-readable medium including computer program instructions encoded thereon that when executed on at least one processor in a computer system causes the processor to perform the operations indicated herein as embodiments of the invention.
Therefore, the invention guarantees a more efficient result thanks to the use of AI instead of just heuristics and a faster processing time thanks to the use of digital maps in rasterized pages. Moreover, the invention is able to deal with geometric distortions on the page; documents including or not text, vectorial PDFs; identify and label both text structures and document elements, such as tables, bullet lists, justified text, image captions, images and graphs, etc.
The previous and other advantages and features will be more fully understood from the following detailed description of embodiments, with reference to the attached figures, which must be considered in an illustrative and non-limiting manner, in which:
Similarly,
That is, the adapter module 18 is used to adapt the document 16 to the required input for the text detector 12. In some embodiments, the input (i.e., the visual representation 10) can be an image format, and the content of this image can be any kind of information that represents the position of the words. It can be the visual page as is or can be another type of representation.
In an embodiment, the use of abstract maps is proposed as visual representation 10. Particularly, these abstract maps are binary images with background to black pixels and words represented as white rectangles.
When the page is in image format, there are multiple manners to obtain the position of the words using an image-based segmentation approach 34. The OCR methods usually return the position of the words, but in this embodiment a method based on projections is preferably proposed since it gives better time responses. Before applying the method based on projections the skew 32 of the image is corrected using a recognition algorithm, for example the Hough Transform.
g(r,c)=G(r,c)*l(r,c),
where the Gaussian filter G is defined as:
The Gradient of g(r,c) is computed as:
M(r,c)=√{square root over (gr2(r,c)+gc2(r,c))},
and
θ(r,c)=tan−1[gc(r,c)+gr(r,c)].
A threshold T is applied to M(r,c):
where T is chosen so that all edge elements are kept while most of the noise is suppressed.
Then, the method suppresses non-maxima pixels in the edges in the MT(r,c) obtained above to thin the edge ridges. To do so, the method checks whether each non-zero MT (r,c) is greater than its two neighbors along the gradient direction θ(r, c). If so, MT (r,c) is unchanged, otherwise, is set to 0.
The previous result can be thresholded by two different thresholds ζ1 and ζ (where ζ1<ζ2) to obtain two binary images T1 and T2. Note that T2 with greater ζ2 has less noise and fewer false edges but greater gaps between edge segments, when compared to T1 with smaller ζ1.
The method links edge segments in T2 to form continuous edges. To do so, the method traces each segment in T2 to its end and then searches for its neighbors in T1 to find any edge segment in T1 to bridge the gap until reaching another edge segment in T2. The output of the Canny Edge Detection algorithm 40 is denoted here as Ic(r,c).
The next steps are morphological operations in order to join and fill the gaps between chars and words and trying to convert text lines into straight lines. This is done at first with a dilate operation 42, and later with an erode operation 44. These morphological operations provides a cleaner image Im(r,c) with straight lines easier to be processed with the Hough Transform.
As the Hough Transform can be slow, the image is preferably resized 46 to work with a lower resolution image (75 dpi works well). This allows achieving the correct angle to de-skew the image at the same time that reduces the time required to get the result.
The Hough transform 50 is applied over the resized image in order to detect the angles of the text lines. The Hough Transform 50 converts the Euclidian space to the Hough Space. The Hough Space is a 2D plane that has a horizontal axis representing the slope and the vertical axis representing the intercept of a line on the edge image. A line on an edge image is represented in the form of y=ax+b (Hough, 1962). One line on the edge image produces a point on the Hough Space since a line is characterized by its slope and intercept b. Usually the Hough Space uses a different form of representing the straight lines called normal lines that passes through the origin and perpendicular to that straight line. The form of the normal line is ρ=x cos(θ)+y sin(θ) where ρ is the length of the normal line and θ is the angle between the normal line and the x axis. In this way each edge point (xi, yi) now generates a cosine curve in the Hough Space instead of a straight line. One line on the edge image still produces a point on the Hough Space. The lines are detected looking for these points in the Hough Space, for this task a threshold Th should be set. If the threshold Th is low then the number of lines obtained will be high, and it could be difficult to determine the more probable direction of the real lines. To avoid this problem, a recursive strategy is proposed. The threshold is set to a high value, if the lines found are less than 10, the process is repeated with a new threshold calculated as Th_i=¾*Th_{i−1}, where i indicates the number of the iteration. Other values different to 10 can be used, but it has been found empirically that this number works really well. When the number of lines is higher than 10, the process is stopped. The stop condition 62 also needs to avoid infinite loops (it is possible to find less than 10 lines in a page), so the method only allows to repeat the process a finite number of times.
In a normal case, after applying the Hough Transform 50, almost ten lines are obtained. The method obtains the vector of angles θs of these lines and calculates the histogram of these angles. The maximum of the histogram 54 is taken as the estimated angle (skew angle) θf.
The de-skew 60 is made just taking −θf and rotating the image. After the rotation, the obtained image Irot(r′,c′) has more width and height than the input image. To correct this distortion the crop operation is applied over the image. An easy way to do that is to calculate the new origin point (r0, c0) as:
where R′ and C′ are the number of rows and columns of the image Irot(r′,c′), and R and C are the number of rows and columns of the input image I(r,c). The crop is done over the image Irot(r′,c′) in the region delimited by the rows [r0, R] and the columns [c0, C].
After this crop operation the output of the skew correction module 32 is obtained; i.e. an image I′(r,c) where the rotation of the page was corrected and with the same size of the original image. This allows to the image-based segmentation achieve a high accuracy.
The performance of this skew correction module 32 was compared with other methods of the state of the art, such as the Leptonica method (Dan S. Bloomberg et al. Measuring document image skew and orientation. In Document Recognition II (pp. 302-316). SPIE), showing better performance in the precision of the angle obtained, the time required to make the estimation and the range of angles that the algorithm can correct.
After correcting the skew of the image, image-based segmentation 34 can be applied. As explained before, a method based on projections can be used. The method starts by binarizing the bitmap format document I′(r,c) and summing the pixels in horizontal direction, the horizontal projection Ph(r) is obtained as:
Ph(r)=ΣcIb(r,c),
where the binarized image is denoted as Ib(r,c), r is the r-th row, c is the c-th column, and I(r,c) is the binarized document image.
By setting a proper threshold value, the bitmap format document can be segmented in text lines, finding optimal binarization thresholds from the image histogram. Once the projections are computed, and text lines are identified, each text line is segmented. For each line, its vertical projection Pvi(c) corresponds to:
P
vi(C)=ΣrIb(r,c),
where, rIib (r,c) is the binarized document image cropped to the i-th line.
Applying the same technique used to isolate text line using the horizontal projection Ph(r), spaces in each line can be identified and measured applying a proper threshold to the vertical projection Pvi(c).
The output of the segmentation 34 is a set of word lengths and a set of their corresponding locations in the document, denoted as set W and set L respectively 38.
If the document is in vectorial format, the system can read the information from the Page Description Language (PDL) based on the standard ISO 32000 known as PDF using a PDF interpreter 36. A PDF document consists of a collection of objects that together describe the appearance of one or more pages, possibly accompanied by additional interactive elements and higher-level application data. The system reads the trailer section of the file, gets the root information, and then reads the pages object. Inside of page objects the contents objects of the page can be found. The contents may have text objects that consist of operators that can show text strings, move the text position, and set text state and certain other parameters. The system is able to read and interpret all text state operators, text-positioning operators, and text-showing operators in order to obtain the set of words lengths and a set of their corresponding locations in the document, denoted as set W and L, respectively. In this case, the operation results faster. The values W and L 38 are used to create a (binary) map image M(r, c) 10 in the binary image creation module 39 where the pixels of the regions associated with the location L and word lengths W are set to white, and the rest of the pixels are set to black. L is a vector of pixel positions {(Ix1,Iy1), (Ix2, Iy2) . . . (Ix_n,Iy_n)} indicating the left-bottom corner of the word, and W is a vector of pair of values indicating the width and high of a word {(w1,h1), (w2,h2) . . . (w_n,h_n)} where the length of the vector L and W are the same since each pair is related to a word of the page. The map M(r,c) is created with the same size as the input image I(r,c) and with all their pixels with value zero (black pixels). Then, for each value i⊂[1,n] the regions delimited by [Li,Li+Wj] are set to white (this is the same as the region [(Ixi,Iyi),(wi,hi)] for all i⊂[1,n]). An example of the input map can be seen on
Once the input to the text detector 12 is generated, in an embodiment, a deep neural network is used to predict in the inference process the regions corresponding to the invalid text. Particularly, the use of a Feature Pyramid Network (FPN) is proposed, although other networks can be equally used such as a Single Shot MultiBox Detector (SSD), among others. The FPN network works well with different scale objects, which can be useful for the data processed here, where the size of the object depends on the size of the list, table, etc. Of course, these models have to be trained to learn the weights of the network before being applied in the inference step. This is done using supervised learning, so the data used should be annotated in previous steps.
A possible implementation of the invention can be based on the use of the Focal Loss and the use of Transfer Learning tasks, using the COCO dataset as a reference. The Focal loss prevents the vast number of easy negatives from overwhelming the text detector 12 during training. The invention can also use the Efficientnet or the Mobilenet architectures as a backbone network in order to reduce the computation terms.
The output of the network will be the set of invalid regions 14 (a set of boxes representing invalid regions) B that represents multiple detections, where each box is defined by {x,y, w, h} values and x indicates the left-up corner in the horizontal coordinate, y indicates the left-up corner of the box in the vertical coordinate, w indicates the width in pixels, and h indicates the height of the box in pixels. The output also returns a score associated to each box. This score indicates the confidence level of the detection (
It should be noted that the problem can be solved as an object detection or an image segmentation problem. In this case, particularly, the detection object scenario is considered. Image segmentation is a further extension of object detection in which the method marks the presence of an object through pixel-wise masks generated for each object in the image. This technique is more granular than bounding box generation because this can help in determining the shape of each object present in the image. This granularity helps in various fields such as medical image processing, satellite imaging, etc., but it is less relevant for the tasks related here, where rectangle bounding boxes are enough to cover the goals of the system since we have corrected the geometric distortion of the pages.
One of the most important contributions of present invention is when these predictions are used as part of a digital text watermarking process based on shifts of the text elements. The solution may also be used for detecting specific objects in text documents, like tables etc., similarly, to state of the art algorithms, but with much faster processing time. The solution can be adapted for different algorithms as unwanted areas are labelled correctly in the train step of the neural network. A specific adaptation can be made using the watermarking method described in EP3477578B1, of the same authors of present invention.
With regard to
The present invention has been described in particular detail with respect to specific possible embodiments. Those of skill in the art will appreciate that the invention may be practiced in other embodiments. For example, the nomenclature used for components, capitalization of component designations and terms, the attributes, data structures, or any other programming or structural aspect is not significant, mandatory, or limiting, and the mechanisms that implement the invention or its features can have various different names, formats, and/or protocols. Further, the system and/or functionality of the invention may be implemented via various combinations of software and hardware, as described, or entirely in software elements. Also, particular divisions of functionality between the various components described herein are merely exemplary, and not mandatory or significant. Consequently, functions performed by a single component may, in other embodiments, be performed by multiple components, and functions performed by multiple components may, in other embodiments, be performed by a single component.
Certain aspects of the present invention include process steps or operations and instructions described herein in an algorithmic and/or algorithmic-like form. It should be noted that the process steps and/or operations and instructions of the present invention can be embodied in software, firmware, and/or hardware, and when embodied in software, can be downloaded to reside on and be operated from different platforms used by real-time network operating systems.
The scope of the present invention is defined in the following set of claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 21382500.3 | Jun 2021 | EP | regional |
