As the use of computers and computer-based networks continues to expand, content providers are preparing and distributing more and more content in electronic form. This content includes traditional media such as books, magazines, newspapers, newsletters, manuals, guides, references, articles, reports, documents, etc., that exist in print, as well as electronic media in which the aforesaid content exists in digital form or is transformed from print into digital form through the use of a scanning device. The Internet, in particular, has facilitated the wider publication of digital content through downloading and display of images of content. As data transmission speeds increase, more and more page images of content are becoming available online. A page image allows a reader to see the page of content as it would appear in print.
Despite the great appeal of providing digital images of content, many content providers face challenges when generating, storing, and transferring the images of content, particularly when the accuracy of recognizing text in images is important. For example, to enable users to read page images from a book or magazine on a computer screen, or to print them for later reading, the images must be sufficiently clear to present legible text, including when scaled to various sizes. Typically, the images are translated into computer-readable data using various character recognition techniques, such as optical character recognition (OCR), which includes digital character recognition. Whether or not OCR is used, a page image may be processed and stored with reference to various glyphs appearing in the page. Glyphs may represent, for example, marks, characters, symbols or other elements appearing in the page. These glyphs may be defined in various ways, including by storing contour or outline information, such as outlines defined by Bezier curves.
One challenge faced by digital content providers is identifying individual glyphs from image data. This may be particularly difficult, for example, when an image includes cursive writing. Another challenge is the cost of storing and transferring glyph-based content. For example, for works written in certain languages, a large number of glyphs are often defined and stored in association with a glyph-based file in order to represent each distinct character or symbol appearing in the work. For glyphs corresponding to text in certain languages, for example, thousands of glyphs may be stored for a given glyph-based file.
The foregoing aspects and many of the attendant advantages will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:
Generally described, aspects of the present disclosure relate to optimizing a glyph-based file for storage, as well as for any subsequent transfer to another computing device. In one embodiment, the glyph-based file may have been generated from an electronic (digital) image containing text, where the image has been scanned and converted into the glyph-based file. One example of a suitable method and system for scanning and converting an electronic image into a glyph-based file for rendering is described in U.S. Pat. No. 7,460,710 entitled “Converting Digital Images Containing Text to Token-Based Files for Rendering,” issued Dec. 2, 2008, which patent is incorporated herein by reference in its entirety. Generally, a glyph may represent a single letter, character, symbol, etc., or may in some cases represent more than one letter, character, symbol, etc. or only a portion of a letter, character, symbol, etc. The “tokens” described in U.S. Pat. No. 7,460,710 may, in some embodiments, be considered glyphs. In other embodiments, glyphs may be associated with a font referenced in a glyph-based file, where each glyph is a letter, character or symbol of the font. Glyphs may be defined and stored in a variety of manners, including many that are well known in the art. For example, in some embodiments, glyphs may be defined with reference to one or more outlines that mathematically describe lines and/or curves representing or bounding a shape. An outline or contour may be stored, for example, in second degree or third degree Bezier curves. In some embodiments, a glyph stored in an outline format may be scaled or transformed, such as by emboldening, rotating, skewing, etc.
Aspects of the present disclosure relate to optimizing or reducing the size of a previously stored glyph-based file, such as by replacing at least a subset of the glyphs referenced in the glyph-based file with composite glyphs that each reference one or more shared components. In some embodiments, a glyph optimization module, as described herein, may optimize a glyph-based file by identifying individual components within glyphs referenced in the glyph-based file. Each identified component within a glyph may be a portion of the glyph. In some embodiments, the identified components of a glyph may include a joint component that is connected to at least one other portion of the glyph, and/or a disjoint component that is not connected to any other portion of the glyph. The glyph optimization module may then determine groupings of components, where the groupings are determined based at least in part by identifying similarly shaped components. The glyph optimization module may then select a representative component from each grouping. The module may then generate composite glyphs and store the composite glyphs in an optimized file, where each composite glyph may include a reference to at least one representative component.
In some embodiments, a glyph-based file server may store an optimized file that replaces at least some of the glyph references in the original glyph-based file with composite glyphs that reference shared components (for example, selected representative components from the groupings). The optimized file may require substantially less memory than the original glyph-based file, especially in the case of a file that includes a number of glyphs with similar portions. The glyph-based file server may then send the optimized file to one or more other computing devices, such as a user device, using less bandwidth than would be needed to send the original file prior to optimization. A computing device may subsequently generate a page image for display based on the optimized glyph-based file, such that the displayed page appears identical or nearly identical to a page generated based on the original glyph-based file. In some embodiments, because less glyph components may need to be stored for a given page image, page images generated based on an optimized file may be displayed using less RAM or other memory than would be needed to display page images generated based on the original glyph-based file. Accordingly, loading time and/or reading speed may be increased on a user device that generates page images from the optimized glyph-based file for display to a user.
As illustrated, the glyph-based file server 110 includes or communicates with a glyph-based file data store 112. Those skilled in the art will appreciate that the glyph-based file data store 112 may be local to the glyph-based file server 110, may be remote to the glyph-based file server 110, and/or may be a network-based service itself. Those skilled in the art will appreciate that the network 108 may be any wired network, wireless network or combination thereof. In addition, the network 108 may be a personal area network, local area network, wide area network, cable network, satellite network, cellular telephone network, etc., or combination thereof. Protocols and components for communicating via the Internet or any of the other aforementioned types of communication networks are well known to those skilled in the art of computer communications and, thus, need not be described in more detail herein.
The memory 210 contains computer program instructions that the processing unit 204 executes in order to implement one or more embodiments. The memory 210 generally includes RAM, ROM and/or other persistent, non-transitory memory. The memory 210 may store an operating system 214 that provides computer program instructions for use by the processing unit 204 in the general administration and operation of the glyph-based file server 110. The memory 210 may further include computer program instructions and other information for implementing aspects of the present disclosure. For example, in one embodiment, the memory 210 includes a user interface module 212 that generates user interfaces (and/or instructions therefor) for display upon a computing device, e.g., via a navigation interface such as a web browser or electronic book reader installed on the computing device. In addition, memory 210 may include or communicate with an auxiliary glyph-based file data store 112. Data stored in the glyph-based file data store 112 may include glyph-based files of various types, as discussed above.
In addition to the user interface module 212, the memory 210 may include a glyph optimization module 216 that may be executed by the processing unit 204. In one embodiment, the glyph optimization module 216 implements various aspects of the present disclosure, e.g., identifying common components of glyphs, grouping components, generating composite glyphs, etc., as described further below. While the glyph optimization module 216 is shown in
In some embodiments, prior to proceeding to block 304, the glyph optimization module 216 may optimize or otherwise alter one or more individual glyphs of the retrieved file prior to identifying components of each glyph (not illustrated). For example, the glyph optimization module 216 may reduce rough edges of the outline(s) of one or more glyphs referenced in the file. Reducing the rough edges may include, for example, decreasing a number of curve points associated with a given glyph. The rough edges may be the result, for example, of artifacts from scanning a page image and/or converting a glyph from some other format. Optimizing an individual glyph prior to block 304 may alternatively or additionally include reducing the number of control points associated with a given glyph. Reducing the number of control points may, for example, reduce the size needed to store the glyph, improve rendering performance and/or improve rendering quality. Optimizing the individual glyphs may also result in a larger number of similarities becoming evident in components or portions of different glyphs of the file, as discussed below.
At block 304, the glyph optimization module 216 identifies disjoint components and/or joint components within glyphs of the retrieved glyph-based file in order to compare the identified components to each other. Depending on the embodiment, the glyph optimization module 216 may identify one or more joint components, one or more disjoint components, or both joint components and disjoint components. A disjoint component may generally refer to a portion of a given glyph that is not connected to any other portion of the given glyph. Disjoint components may be identified, for example, by analyzing the glyph data for a given glyph to locate two or more portions or subsets of the glyph that do not connect to each other. A joint component, in contrast, may generally refer to a portion of a given glyph that is connected to at least one other portion of the given glyph. Despite being connected to another portion of a given glyph, a joint component may nonetheless be determined, using methods discussed below, to be identical or similar to a component of another glyph. Examples of joint components and disjoint components are discussed in more detail below with reference to
In embodiments in which the glyph optimization module 216 identifies joint components at block 304, the joint components may be identified within a given glyph, for purposes of comparing the joint component to components of other glyphs, using a variety of methods. For example, a portion of a glyph may be identified as a distinct joint component based at least in part by carving the glyph. Carving the glyph may include algorithmically eroding the glyph until a portion of the glyph disconnects from at least one other portion of the glyph. The disconnected portion may then be identified as a joint component to be compared to components of other glyphs, as described further below. The eroded glyph, or a composite glyph that includes one or more eroded components, may later be re-grown using a corresponding algorithm in order to generate a glyph matching the original glyph. Another method for identifying a joint component of a glyph may include identifying one or more continuous strokes within the glyph. Another method may include locating lines or strokes that connect through a central point or central line within the glyph. Yet another method may include searching the glyph data for portions of glyphs that match one or more previously identified components and/or one or more previously stored glyphs. In some such embodiments, identifying components at block 304 may be implemented as part of, or in parallel with, block 306 below, where the glyph optimization module 216 identifies and groups similar components.
Next, at block 306, the glyph optimization module 216 determines one or more component groupings of the components identified in block 304. The groups or groupings may be determined based at least in part by identifying similarities between components of different glyphs. As discussed above, in some embodiments, components of the glyphs may be identified during the grouping process, either instead of or in addition to components being identified prior to block 306. In some embodiments, the components identified at block 304 (which may include joint components and/or disjoint components) may generally each be considered candidate components for comparison with other glyphs' components in order to identify shared components among the glyphs of the file.
In some embodiments, grouping components may include identifying similarly shaped components and/or forming clusters of components based on shape. As will be appreciated by those of skill in the art, any of a number of different methods may be used to compare glyph components. In some embodiments, identifying similar components may include applying a relatively fast comparison method that serves as a coarse filter, then a relatively slower comparison method that serves as a finer filter to compare components that have met a given threshold using the coarse filter method. An example of a coarse filtering method to determine similarity between two components may be, for example, comparing descriptors for the components. A descriptor may be, for example, a shape context descriptor that includes an array of values that generally represent a shape. For example, a shape context descriptor may reference the number of contour pixels that fall into each of a number of bins, but which do not capture all of the detail or information needed to fully render or display the glyph (or glyph component). Bins may be defined, for example, by a certain number of distances and orientations that form a circular grid or other pattern that divides a space in which the glyph or component may be rendered. The same bin pattern may be compared to each component, and the number of contour points or pixels that fall into each bin for each component may be used to determine a shape context descriptor for the given component. As will be appreciated, shape context descriptors may be defined in a number of ways that may each result in different values that generally describe or represent the same shape. Accordingly, depending on the embodiment, any of a number of different methods may be used to determine shape context descriptors, but the same method should generally be used to determine the shape context descriptor for each component to be compared in the given embodiment. Once a descriptor is generated for each identified component, the descriptor values for each component may be compared to the descriptor values for each other component in order to identify groups of components that fall within a given threshold of distance between descriptor values. In some embodiments, the threshold may be set to be relatively small, such that tight groups or clusters are formed. In embodiments in which there is less desire or need to generate composite glyphs that are very similar to the original glyphs, a lower threshold may be used, resulting in looser cluster formation.
Another method that may be used to compare components, which may be considered a finer filter method than the shape context descriptor comparison discussed above, is to directly compare shapes as they would be rendered or displayed. For example, such a method may include comparing pixel data or other image data of the identified components. In some embodiments, the image data may only be compared for shapes or components that have passed a first, faster comparison method (such as a descriptor comparison). In other embodiments, the glyph optimization module 216 may compare the image data of each component to each other component without first applying a coarser filter.
The glyph optimization module 216 may group similar components, for example, by clustering the components based on one or more of the methods discussed above, such that each component in any given cluster is within a predetermined threshold of any other component of the cluster. In one embodiment, for example, the components' descriptors may be clustered based on a “nearest neighbor” approach. In some embodiments, when determining clusters and/or comparing descriptors, the glyph optimization module 216 may apply a transformation to one or more of the components, and/or may compare the components without reference to transformation data. For example, in some embodiments, each component may be stored with reference to an x value, a y value, and a transformation value. In some such embodiments, components may be compared such that components with different transformation values may nonetheless be considered similar to each other.
Once the glyph optimization module 216 has formed the groups or clusters of components, the illustrative method 300 proceeds to block 308, in which the glyph optimization module 216 selects a representative component from each group. The representative component may be used to replace each component of the group in the optimized file, as discussed further below. Accordingly, it may be desirable to select a representative component that represents a typical or average component from the group. In some embodiments, each group or cluster of components may be sorted based on shape descriptor values, and the representative component for each grouping may be selected to be the component closest to the center of the cluster (e.g., the component associated with the median shape descriptor value of the cluster). In other embodiments, the representative component may be selected based on other methods, such as determining the component that has the smoothest edges, least control points, and/or which requires the least amount of memory to store.
Next, at block 310, the glyph optimization module 216 generates composite glyphs, where each composite glyph references at least one representative component of a group. The composite glyphs may generally replace a corresponding glyph from the original glyph-based file when generating and/or storing the optimized file. As an example, consider a glyph from the original retrieved file that has been determined to comprise two components that have been grouped into two different groups by the glyph optimization module 216, based on the methods described above. A composite glyph to replace this glyph may be generated, where the composite glyph is defined with reference to the two representative components selected for the given two groups, rather than the glyph being defined with the full glyph description previously stored in the retrieved file. The composite glyph may additionally include information other than the references to the components comprising the glyph, such as a transformation value and/or location information regarding the glyph's location within the page image. Accordingly, in some embodiments, the composite glyph may be stored with sufficient information to render the glyph with substantially the same appearance as the originally stored glyph, but in a manner such that glyph components that frequently occur in different glyphs might only be defined once in the optimized glyph-based file.
In some embodiments, a composite glyph may be generated for each glyph of the retrieved file that includes a component that appears in any other glyph of the file. In other embodiments, representative components and composite glyphs referencing those representative components may only be stored when certain criteria is met. As one example, the glyph optimization module 216 may consider the storage space that would be saved by replacing a given group of components with references to a representative component of the group. For example, in some embodiments, the glyph optimization module 216 may store a representative component, as well as composite glyphs referencing the given representative component, when at least a certain minimum number of composite glyphs would reference the given representative component. In other embodiments, the glyph optimization module 216 may generate and store a composite glyph to replace any glyph that contains at least a minimum number of components that are similar to components of at least one other glyph.
Once the glyph optimization module 216 has generated composite glyphs at block 310, the illustrative method proceeds to block 312. At block 312, the glyph optimization module stores an optimized file including the generated composite glyphs in one or more data stores, such as glyph-based file data store 112. In some embodiments, the glyph optimization module 216 may store a new optimized glyph-based file at block 312. In other embodiments, the glyph optimization module may modify the original glyph-based file to replace at least a subset of the glyphs with composite glyphs. The optimized file may generally include some glyphs stored as they were in the original file (such as glyphs for which no composite glyph was generated), as well composite glyphs in place of at least a subset of the original glyphs from the retrieved file. Each composite glyph may be defined, for example, with reference to the one or more components making up the glyph, as well as additional information to be used to reconstruct the glyph (such as transformation information, location information, etc.). The shared components (e.g., representative components discussed above) may be stored, for example, in a shared component table associated with the optimized file. After storing the optimized file, the illustrative method ends at block 314.
As further illustrated in
It is to be understood that not necessarily all objects or advantages may be achieved in accordance with any particular embodiment described herein. Thus, for example, those skilled in the art will recognize that certain embodiments may be configured to operate in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other objects or advantages as may be taught or suggested herein.
All of the processes described herein may be embodied in, and fully automated via, software code modules executed by one or more general purpose computers or processors. The code modules may be stored in any type of computer-readable medium or other computer storage device. Some or all the methods may alternatively be embodied in specialized computer hardware. In addition, the components referred to herein may be implemented in hardware, software, firmware or a combination thereof.
Conditional language such as, among others, “can,” “could,” “might” or “may,” unless specifically stated otherwise, are otherwise understood within the context as used in general to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment.
Conjunctive language such as the phrase “at least one of X, Y and Z,” unless specifically stated otherwise, is otherwise understood with the context as used in general to convey that an item, term, etc. may be either X, Y or Z. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y and at least one of Z to each be present.
Any process descriptions, elements or blocks in the flow diagrams described herein and/or depicted in the attached figures should be understood as potentially representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or elements in the process. Alternate implementations are included within the scope of the embodiments described herein in which elements or functions may be deleted, executed out of order from that shown, or discussed, including substantially concurrently or in reverse order, depending on the functionality involved as would be understood by those skilled in the art.
It should be emphasized that many variations and modifications may be made to the above-described embodiments, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.
Number | Name | Date | Kind |
---|---|---|---|
4173753 | Chou | Nov 1979 | A |
4251871 | Yu | Feb 1981 | A |
4490789 | Leban et al. | Dec 1984 | A |
4511267 | Pokorny et al. | Apr 1985 | A |
4566128 | Araki | Jan 1986 | A |
4621340 | Pokorny et al. | Nov 1986 | A |
4670841 | Kostopoulos | Jun 1987 | A |
4723217 | Nakano et al. | Feb 1988 | A |
4739318 | Cohen | Apr 1988 | A |
4850026 | Jeng et al. | Jul 1989 | A |
5109352 | O'Dell | Apr 1992 | A |
5305207 | Chiu | Apr 1994 | A |
5444840 | Froessl | Aug 1995 | A |
5468077 | Motokado et al. | Nov 1995 | A |
5574842 | Takakura | Nov 1996 | A |
5586198 | Lakritz | Dec 1996 | A |
5586241 | Bauermeister et al. | Dec 1996 | A |
5673064 | Seto | Sep 1997 | A |
5727140 | Ohtomo et al. | Mar 1998 | A |
5768451 | Hisamitsu | Jun 1998 | A |
5831636 | Merchant et al. | Nov 1998 | A |
5835100 | Matsufusa | Nov 1998 | A |
5903904 | Peairs | May 1999 | A |
5917501 | Muller et al. | Jun 1999 | A |
5943443 | Itonori | Aug 1999 | A |
5969727 | Kaneko | Oct 1999 | A |
5982387 | Hellmann | Nov 1999 | A |
6003049 | Chiang | Dec 1999 | A |
6144765 | Tamura et al. | Nov 2000 | A |
6181353 | Kurisu | Jan 2001 | B1 |
6341176 | Shirasaki | Jan 2002 | B1 |
6483510 | Jeong | Nov 2002 | B1 |
6501475 | Cheng | Dec 2002 | B1 |
6661417 | Cheng | Dec 2003 | B1 |
6697524 | Arai et al. | Feb 2004 | B1 |
6704116 | Abulhab | Mar 2004 | B1 |
6748115 | Gross | Jun 2004 | B1 |
6766179 | Shiau et al. | Jul 2004 | B1 |
6771267 | Muller | Aug 2004 | B1 |
6867787 | Shimizu et al. | Mar 2005 | B1 |
6879951 | Kuo | Apr 2005 | B1 |
6882344 | Hayes et al. | Apr 2005 | B1 |
6947771 | Guo et al. | Sep 2005 | B2 |
6952210 | Renner et al. | Oct 2005 | B1 |
6956968 | O'Dell et al. | Oct 2005 | B1 |
6992671 | Corona | Jan 2006 | B1 |
7003158 | Bennett et al. | Feb 2006 | B1 |
7009612 | Hakamada | Mar 2006 | B2 |
7012605 | Manome | Mar 2006 | B1 |
7251365 | Fux et al. | Jul 2007 | B2 |
7263658 | Chou | Aug 2007 | B2 |
7443400 | Matskewich et al. | Oct 2008 | B2 |
7539939 | Schomer | May 2009 | B1 |
7573476 | Matskewich et al. | Aug 2009 | B2 |
7612897 | Hodder | Nov 2009 | B2 |
7710422 | Matskewich et al. | May 2010 | B2 |
7787694 | Fux et al. | Aug 2010 | B2 |
8243077 | Cheng | Aug 2012 | B2 |
8290269 | Wu | Oct 2012 | B2 |
8295600 | Wu | Oct 2012 | B2 |
8352855 | Levy | Jan 2013 | B2 |
20020064311 | Yahagi | May 2002 | A1 |
20020085006 | Shade et al. | Jul 2002 | A1 |
20030027601 | Guo et al. | Feb 2003 | A1 |
20030043151 | Choi et al. | Mar 2003 | A1 |
20040006749 | Fux | Jan 2004 | A1 |
20040012591 | Ito et al. | Jan 2004 | A1 |
20040148577 | Xu et al. | Jul 2004 | A1 |
20040155882 | Wu | Aug 2004 | A1 |
20040227771 | Arnold et al. | Nov 2004 | A1 |
20050106537 | Chepaitis | May 2005 | A1 |
20050162430 | Stamm et al. | Jul 2005 | A1 |
20050174314 | Furihata | Aug 2005 | A1 |
20060017733 | Matskewich | Jan 2006 | A1 |
20060079281 | Ravindra et al. | Apr 2006 | A1 |
20060095843 | Chou | May 2006 | A1 |
20060181532 | Ravindra et al. | Aug 2006 | A1 |
20060238539 | Opstad | Oct 2006 | A1 |
20060256116 | Burago et al. | Nov 2006 | A1 |
20070139415 | Stamm et al. | Jun 2007 | A1 |
20070154094 | Lin et al. | Jul 2007 | A1 |
20070160292 | Wu | Jul 2007 | A1 |
20070189628 | Nolan et al. | Aug 2007 | A1 |
20080030502 | Chapman | Feb 2008 | A1 |
20080063278 | Vincent | Mar 2008 | A1 |
20080063279 | Vincent | Mar 2008 | A1 |
20080068383 | Dowling | Mar 2008 | A1 |
20080068384 | Achong et al. | Mar 2008 | A1 |
20080100623 | Gurcan | May 2008 | A1 |
20080170810 | Wu | Jul 2008 | A1 |
20080193015 | Hong | Aug 2008 | A1 |
20080218522 | Fux et al. | Sep 2008 | A1 |
20080221866 | Katragadda et al. | Sep 2008 | A1 |
20080240567 | Chaoweeraprasit et al. | Oct 2008 | A1 |
20080306916 | Gonzalez | Dec 2008 | A1 |
20090028435 | Wu et al. | Jan 2009 | A1 |
20090097765 | Kimura et al. | Apr 2009 | A1 |
20090153471 | Lee | Jun 2009 | A1 |
20090310868 | Oota | Dec 2009 | A1 |
20090324082 | Liu et al. | Dec 2009 | A1 |
20100039916 | Hasegawa et al. | Feb 2010 | A1 |
20100053171 | Cheng | Mar 2010 | A1 |
20100174732 | Levy | Jul 2010 | A1 |
20100238473 | Tanaka | Sep 2010 | A1 |
20100245362 | Huang | Sep 2010 | A1 |
20110090230 | Bacus et al. | Apr 2011 | A1 |
20110093565 | Bacus et al. | Apr 2011 | A1 |
20110141029 | Fux | Jun 2011 | A1 |
20110286662 | Lai | Nov 2011 | A1 |
20110298719 | Wong | Dec 2011 | A1 |
20120069027 | Yamazaki et al. | Mar 2012 | A1 |
20120081297 | Heo | Apr 2012 | A1 |
20120089632 | Zhou et al. | Apr 2012 | A1 |
20120092345 | Joshi et al. | Apr 2012 | A1 |
20120235920 | Fux | Sep 2012 | A1 |
20120331383 | Park | Dec 2012 | A1 |
20130057554 | Linnerud | Mar 2013 | A1 |
20130113806 | Naveh | May 2013 | A1 |
20130120396 | Kaplan | May 2013 | A1 |
20130127872 | Kaplan | May 2013 | A1 |
20130155098 | Rickner et al. | Jun 2013 | A1 |
20150139559 | Smith | May 2015 | A1 |