Patent Application
20090169061
The embodiments of the invention relate to a reading machine for impaired persons such as individuals who are visually impaired or have dyslexia.
People with disabilities, such as impaired vision or dyslexia, may have difficulty reading printed material. Automatic systems are needed to render documents as audio recordings.
It is known to provide a mobile print digitizer for the visually impaired. One known device captures printed documents and reads them to the user. A camera or scanner captures an image of a printed page, and then runs optical character recognition (OCR) on the image. The output is fed to a speech synthesizer such as a text-to-speech system (TTS). Unfortunately, existing systems can be inefficient in their ability to allow a user to efficiently navigate a document as the reader reads it to the user. Accordingly, new approaches are desired.
Embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.
One of the challenges for users of reading devices is that with audio playback, textual hierarchy (e.g., outline organization) may be lost that would otherwise be conveyed, e.g., via size and formatting seen by visual users viewing the text. Accordingly, in some embodiments, a reading apparatus with an auditory presentation that preserves at least some of the intended hierarchy of information is provided. For example, in some embodiments, the user can navigate a given text by skipping through chunks of text delineated by differences in font size or weight, or commonly recognized words or symbols that indicate natural break points.
The processor and memory may comprise any suitable combination of memory and processing circuits, components, or combinations of the same to execute routines to control the reader 102. The memory 110 comprises device control (DC) software code 111 to control the reader 102 and execute its various functions. In the depicted embodiment, the device control code has several modules including optical character recognition (OCR) and text-to-speech (TTS) modules. The OCR module further includes a characterization (CZN) module. There may be more modules and in some embodiments, the modules may not necessarily be related to each other as shown.
The device control program controls scanning (digitized document acquisition), reading navigation, and general system functionality. The OCR module converts the pre-text (e.g., rasterized scanned image) document into text data, characterizes it, and reads it to a user through a convenient navigation interface. (As used herein, “reading” means to convey or provide text in an audio form to a user.)
The camera may comprise any suitable device such as a charge coupled device (CCD) camera to acquire a raster image of the text document 101, as is known in the art. It may scan a document line by line, section by section or it may image an entire page or sheet at once. Likewise, the auditory device 112 could comprise any suitable device to auditorily convey the read text to the user. For example, it could comprise one or more speakers and/or audio interface ports for connection to headphones or the like.
The user interface 106 may constitute any suitable components, known or not yet developed, to allow a user to conveniently control the reader. For example, the user interface could comprise one or more buttons, wheels, joysticks or other input control components that allow a user to manually control the reader without necessarily being able to see the user interface (i.e., control the components by feeling them). In some embodiments, the user interface includes a five button interface, such as that shown in
The user interface could also include input ports (which may also function as output ports) such as universal serial bus (USB), so-called “Firewire”, and/or wireless ports, e.g., to allow a user to import an electronic document that may or may not be in a text format. For example, portable document format (PDF) documents (or the like) could be imported for auditory reading to a user. In addition, the user interface could include speech-to-text capability, e.g., a microphone with suitable speech-to-text engine. Of course, as with any feature, especially those requiring substantial hardware and/or processing, trade-offs must be made between cost, power consumption, operating efficiency, performance accuracy, and feature capability.
At 201, a digitized document with text to be read is acquired. This may be done through the importation of an electronic document or by scanning a document such as text-containing document 101 using the camera 108. The digital document file should be in a suitable form for the utilized OCR module. For example, many OCR packages typically accept raster image formats commonly found throughout the document management industry such as TIF, BMP, PCX and DCX, to mention just a few. Depending upon particular design considerations, a suitable OCR solution may be used to accept input from other sources such as fax input formats, PDF (or similar) formats, or common scanner driver formats such as TWAIN and ISIS.
At 203, optical character recognition (OCR) and characterization is performed on the acquired document file. Any suitable OCR tool (or module), presently available or specifically developed, capable of suitably identifying text layout and format attributes may be employed. Currently available OCR tools are generally flexible enough to conform to needed performance for techniques discussed herein.
OCR modules generally perform text recognition by defining an area within a frame to be converted and then processing it and examining the results. They typically define vector bounding boxes around blocks or sections of text such as individual characters, sentences, lines, paragraphs, etc. This is illustrated in
OCR programs are usually fairly flexible in allowing designers to adjust software imaging attributes. Varying image attributes may enhance the manner in which the OCR software views text. For example, lowering the software, or display, resolution (e.g., not the scanned resolution) may allow the software to “see” a clearer image of the text thus improving the initial chances of correct recognition. Configuration settings may also be varied based on given design considerations. They can affect the format characteristics of what is being read such as text style (e.g., plain, italic, numeric, or image), text size, and font type.
Most OCR software allows the user to set margins of acceptable error when attempting to recognize a text image. Similarly, confidence levels for recognition depending on results of first iterations may be used and adjusted depending upon design considerations. Confidence levels are measures of certainty. Depending upon desired operating performance, different (e.g., two or more) different confidence levels could be used.
Error detection methodologies are also commonly employed. For example, error detection tools include spell checks and automatic corrections based on the software or user specific dictionaries. Various properties such as uppercase words, numeric expressions, roman numerals, proper nouns, math functions, abbreviation and acronyms may also be compared against appropriate more-particular dictionaries. Resolution of patterns involving text and numerals may be varied according to ambiguities dominant in the patterns.
With embodiments discussed herein, characterization, possibly among other things, involves characterizing OCR'd text blocks so that they may be read using hierarchy to allow a user to more efficiently navigate desired text to be read. More on this will be discussed with reference to
Next, at 205, the OCR'd and characterized text is read to the user. Any suitable text-to-speech (TTS) solution may be used. In some embodiments, the user is allowed to navigate through the text sequentially or hierarchically. For example, in a hierarchal mode, with the user interface of
Distinguishing between main and ancillary text blocks can allow for more efficient sequential reading capabilities. For example, a user might select to read sequentially only the main text in an article. In this way, ancillary text such as figure descriptors or footnotes could be referenced but not read so that the main text of the article could be more efficiently ingested.
(As used herein, the term “article” refers generally to any distinct text subject. For example, an article could be an actual article in a magazine or newspaper, or it could be a menu, a whole book, a prescription tag, a bill, or a receipt.)
Main and ancillary text blocks can be identified in any suitable way and will likely involve application of several different criterions. For example, continuity content analysis could be used. Alternatively or in addition, simpler characteristics such as relative layout and font characteristics, along with simple heuristics and/or assumptions could be applied. For example, Text blocks proximal to an image but different from a majority of text on a page could be presumed to be figure description text. Other types of ancillary text blocks could be identified in ways more particular to their specific attributes. For example, footnote references usually are numeric and are located in a superscript relationship at the end of a sentence or word. When a footnote is identified, the likelihood that its encompassing text block is a main text block is increased (i.e., may assume encompassing block is main text). In addition, the routine will know to “look” for a corresponding numeric (probably at the lower end of the page) leading a block of text that will likely be smaller than the encompassing text. It could then be reasonably assumed that this block(s) is ancillary text (and associated with the footnote reference). Other combinations of suitable criterion and/or assumptions could be used for similar or different types of ancillary (or main) text, and the invention should not be so limited.
At 304, the ancillary text is associated with appropriate main text blocks and/or with appropriate image or other types of blocks. This involves linking the ancillary text with the associated block so that it can be read if prompted or in a read mode where ancillary text is read. It also may involve linking (e.g., sequentially) the ancillary text with other related ancillary text. For example, byline type information such as author information, accreditation information, etc., could be linked as ancillary text to a common main title text block.
Ancillary text blocks may be associated with main text or other block types in any suitable way. As with other aspects of characterization, it may take into account (or even be driven by) the layout of the page. For example, the routine may follow printing rules, e.g., with magazine, journal, book specific printing rules. As already discussed, it should also consider unique characteristics of already identified blocks. For example, image blocks likely have associated descriptive ancillary blocks, footnote reference numbers will likely have associated footnote text, and a text block, aligned with an identified title block may likely be ancillary byline text.
At 306, hierarchy levels are assigned to main text blocks. The assignments are made at least based on text format attributes. Any suitable attributes and/or filters can be used to assign different relative levels. For example, in some embodiments, text that is 10% larger or greater than the majority of text in a different block could be tagged as a higher level. For a given article, there may be many different sizes and types, so alignment and style could also be taken into account. The content of the text itself could also be considered.
For example, certain phrases may be identified as being more likely to be a header or title, rather than a narration such as in an article paragraph. Actual key words could also be recognized. For example, in a book, the word “Chapter n” could be interpreted as a chapter break, especially if not surrounded by words or if larger and/or with a different font than words proximal to it.
Headers, for example, could be detected by the combination of different size, font, and lack of ending punctuation. Magazine editors are currently fond of splashing text in the middle of articles, functioning as a skimming aid or attention-getter, which varies in size or color. Sometimes two or three sizes may be used with one of these sentences, as a way to grab attention. Sometimes the size changes are counter-intuitive (smaller text used to grab attention). Therefore, position of the text in the block could be a factor in the presentation, with text in a similar font, but different than the article body being called out as a heading.
(The dotted arrows in the flow diagram are meant to indicate that the depicted block processes do not necessarily occur sequentially. That is, they may be running concurrently with each other and be affected by characterizations, identifications, and OCR'ing that has already occurred. They also may affect how each other processes its tasks and certainly can affect how OCR'ing and even digitized document acquisition is occurring. For example, as already discussed, when a footnote reference is identified at block 306, this information may be made available to 302 and 304, which could influence them in characterizing an appropriate block, e.g., near bottom of page with leading matching numeric, to be characterized as associated ancillary text. As another example, in any aspect of characterization, the routine might instruct the OCR process to increase software resolution in a certain area if certain text is “expected” but not yet identified. Accordingly, it should be appreciated that the routine blocks and processes discussed herein may be performed in various orders and will likely affect each other, e.g., sequentially, iteratively, and/or otherwise.)
At 323, sequential identification is assigned to the blocks. This may comprise ordering the blocks (indicated with the vector bounding boxes) based on their relative positions to one another. For example, as with the examples of
At 324, the blocks are identified as main, ancillary, navigational, or other. Main and ancillary text may be identified as discussed above. Navigational text is text such as page numbers, column numbers, page or article continuation references (e.g., “continued from page 23), and the like. They may be used to more efficiently piece together a sequence of main text blocks for a given article.
At 326 and 330, ancillary block association and main block level assignments, as discussed previously, are performed. At 328, navigational text blocks are associated with other relevant blocks. For example, they may be associated with main or ancillary text (e.g., page numbers associated with narrative text, footnotes, etc. on the page) and/or they may be linked with other navigation blocks. For example, page numbers would likely be linked with previous and/or subsequent page numbers.
Likewise, the ancillary blocks have association identifiers. For example, ancillary byline block 404 has an identifier of “A[1]:1”, whereby the brackets identify the block from which it is associated (in this case, block 1), while the “1” after the “:” indicates that it is the first ancillary block associated with block 1 (also referenced as 402).
In the preceding description, numerous specific details have been set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known circuits, structures and techniques may have not been shown in detail in order not to obscure an understanding of the description. With this in mind, references to “one embodiment”, “an embodiment”, “example embodiment”, “various embodiments”, etc., indicate that the embodiment(s) of the invention so described may include particular features, structures, or characteristics, but not every embodiment necessarily includes the particular features, structures, or characteristics. Further, some embodiments may have some, all, or none of the features described for other embodiments.
The invention is not limited to the embodiments described, but can be practiced with modification and alteration within the spirit and scope of the appended claims.
It should be appreciated that example sizes/models/values/ranges may have been given, although the present invention is not limited to the same. As manufacturing techniques (e.g., photolithography) mature over time, it is expected that devices of smaller size could be manufactured. Further, arrangements may be shown in block diagram form in order to avoid obscuring the invention, and also in view of the fact that specifics with respect to implementation of such block diagram arrangements are highly dependent upon the platform within which the present invention is to be implemented, i.e., such specifics should be well within purview of one skilled in the art. Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the invention, it should be apparent to one skilled in the art that the invention can be practiced without, or with variation of, these specific details. The description is thus to be regarded as illustrative instead of limiting.
