Patent Application
20150097879
This invention generally relates to an electronic display. The invention also relates to methods and apparatus for processing images to be displayed on the electronic display.
There are various types of electronic displays, for example reflective displays such as electrophoretic, electrowetting, electrofluidic and photonic, or emissive displays such as LCD. Such electronic displays may be incorporated in an electronic document reader which is a device such as an electronic book which presents a document to a user on a display to enable the user to read the document.
When power is removed from emissive displays (such as LCD, OLED and Plasma) they revert to an off-state. This state is known and any colour can be driven accurately from this starting point. Reflective displays, e.g. electrophoretic displays, differ since they retain the last image that was written to them. Therefore, the display must be unwritten before it is rewritten. An electrophoretic display is a display which is designed to mimic the appearance of ordinary ink on paper and may be termed electronic paper, e-paper and electronic ink. Electrophoretic display media is unlike most display technologies.
Typically the image displayed on an electrophoretic display is greyscale (or monochrome). Displaying coloured documents using a black and white display often results in the loss of important information. Colours that were used to distinguish different parts of content can be rendered to grey levels that are so similar that it is difficult to tell the difference. Similarly, coloured text may be converted to a grey level so light it makes it difficult to read.
The table illustrates the maximum contrast ratio, number of unique colours and typical resolution of some information displays and printed media.
As set out above, e-paper display displays have a unique challenge over some other display technologies; they neither support the number of colours that an LCD has, neither do they have the resolution that printed media have to enable efficient “half toning” or “dithering”. When displaying content, originally designed for colour display or print, these deficiencies can lead to a degraded user perception of quality, and in the worst case information can easily be lost.
These challenges are not unique to e-paper displays and there are other displays having limited colour gamut, limited number of colours/shades of grey and/or limited dynamic range and/or contrast. Where these challenges exist, the applicant has thus recognised the need for an improved display, particularly but not limited to an electrophoretic display. Such displays are termed “limited colour” displays. The improvement may relate to the processing of data representing the image which may be done within an electronic document reader itself or in a separate electronic device, e.g. a laptop, mobile phone etc.
According to a first aspect of the invention, there is provided a method of driving a display having an array of pixels, a driver for driving each of said pixels in said array and a colour filter which is aligned with said display whereby each of said pixels is sub-divided into a plurality of sub-pixels of different colours, the method comprising
The step of generating a brightness image may be considered to be encoding the brightness information. By generating a brightness value for each sub-pixel, the generating step may be considered to be encoding the brightness information at full resolution. The step of generating an output signal having an output value for each of the sub-pixels may be considered to overlaying the colour resolution. In other words, the content may be considered to be rendered to monochrome resolution in a first step and the colour filter is “multiplied” over the top as a second step. Thus, the brightness image is generated without considering the colour which is required to represent the target image. Once the brightness image has been generated, generating the output signal comprises determining whether or not a particular sub-pixel is required to generate the required colour. The output signal is generated from the brightness image.
Generating the brightness image may comprise overlaying the target image with a grid (or matrix) having a plurality of cells with each cell corresponding to one of the plurality of sub-pixels within the colour filter.
The brightness value may be set to a value representing black if the sub-pixel (or cell within the grid corresponding to the sub-pixel) covers less than a threshold amount of said target image. The brightness value may be set to a value representing white or grey if the sub-pixel (or cell) covers more than a threshold amount of said target image. White may represent full brightness and partial brightness by grey. There may be a plurality of shades of grey to represent a plurality of states of brightness. Said threshold amount may be 50%.
The brightness image may be a waveform which may be a set of transitions telling each pixel how to change from one state to the next state.
Said output signal may define a sub-pixel mask for each of said different colours wherein each sub-pixel mask comprises the output value for each sub-pixel of the same colour. Each of said plurality of pixels may be divided into four sub-pixels, for example red, green, blue and white. Other colour filters are known and these may be used.
Said output image may be defined as:
Out(i,j)=Rm(i,j)·I(i,j,R)+Gm(i,j)·I(i,j,G)+Bm(i,j)·I(i,j,B)+Wm(i,j)·I(i,j,W)
i,j are the co-ordinates in rows and columns of the pixel array,
Rm(i,j), Gm(i,j), Bm(i,j), Wm(i,j) are red, green, blue and white pixel sub-masks, and
I(i,j,R), I(i,j,G), I(i,j,B), I(i,j,W) are red channel, green channel, blue channel and white channel for the target image respectively.
The output value may be set to zero when the sub-pixel is not required to create target image. A sub-pixel may be determined to be not required either as a result of the determining of the brightness image, i.e. by being set to black, or as a result of the determining the output signal. In the latter determining step, where a single colour corresponding to one of the filter colours, e.g. red, is to be created, determining the output signal is relatively straightforward because only the sub-pixels for that colour are required to create the colour for the whole pixel. However, where the required colour is a combination of or a lighter version of some of the filter colours, determining the output signal will be more complicated and will require a combination of the sub-pixels to create the desired effect for each pixel. Thus, generating the output signal may comprise determining the brightness value of each sub-pixel within the brightness image and the determining the overall colour for each pixel which is required to recreate the colour within the target image.
According to another aspect of the invention, there is provided a method of driving a display, the method comprising
Dividing the target image into a plurality of layers, e.g. dark text, light text, background colour etc., allows the optimisation of the rendering of each layer. Each layer may have only similar content or alternatively multiple types of content may be grouped into a layer whereby similar processing techniques are to be applied to that layer. In other words, dividing said target image may comprise dividing said target image into a plurality of types of content. Each of said plurality of layers may comprise a different type of content. The different types of content may comprise at least two of dark text, light colour text, colour blocks, images and user interface elements. The different type of content may be determined by determining an optimisation technique which generates an output layer signal with optimised display for each type of content and grouping types of content having similar optimisation techniques. It will be appreciated that each group may have one or several types of content. Each of said plurality of layers may thus comprise a different type of content each having a similar optimisation technique. Thus, generating the output layer signal may comprise applying the appropriate optimisation technique to each layer.
It will be appreciated that this aspect may be combined with the previous aspect by dividing said target image into said plurality of layers and generating the brightness layer and output signal for each layer separately. The following features apply to both aspects of the invention.
Thus according to another aspect of the invention, there is provided a method of driving a display, the method comprising
Said dividing step may comprise defining of a dark text layer comprising predominantly dark text, a light coloured text layer comprising predominantly light coloured text, a colour block layer comprising predominantly blocks of colour, an image layer comprising predominantly images, and a user interface layer comprising predominantly user interface elements. Predominantly means that most of the layer comprises the specified features only although it will be appreciated that there may be some overlap between the features.
Said generating step may comprise optimising the text colour by setting all dark text to black. By dark text, it is meant black text, dark grey or dark blue text, or any similar colour text which is close to black. Said generating step may further comprise generating an output layer signal as a fast waveform which drives the display to produce the text before other elements. By “fast” waveform it is meant that the text appears first on the display and this is achieved by a simple waveform.
For other colours of text, said generating step may comprise optimising the text colour by comparing said text colour to a table of colours and replacing the text colour with a closest matching colour within said table of colours. Similarly, said generating step may comprise optimising the block colour by comparing said block colour to a table of colours and replacing the block colour with a closest matching colour within said table of colours.
For said image layer, said generating step may comprise standard optimisation effects, e.g. sharpening images, saturation boosting. Said generating step may comprise generating an output layer signal as an accurate waveform which drives the electrophoretic display to produce the image after other elements. A more accurate waveform may comprise more and varied transitions, e.g. to each of the greys.
For the user interface elements, said generating step may comprise generating an output layer signal as a transition waveform which drives the electrophoretic display to create an illusion of movement. “Transition waveforms” may be defined as waveforms that combines not only grey-level to grey-level information, but some spatial rules about the order in which the pixels are updated.
According to another aspect of the invention, there is provided a method of driving a display, the method comprising
As before, the above aspect may be combined with other aspects, for example, by converting one or more layers to greyscale and carrying out the comparing step.
The above methods may be implemented in many types of display, particularly those having one or more of the following problems:
Thus according to another aspect of the invention, there is provided a display having an array of pixels, a driver for driving each of said pixels in said array, wherein said driver comprises an input for receiving said output signal or said composite output signal described above. The display may be an emissive, e.g. an electrophoretic display. The display may be incorporated in an electronic document reader. The electronic document reader may further comprise a colour filter which is aligned with said display whereby each of said pixels is sub-divided into a plurality of sub-pixels of different colours
The electronic document reader may further comprise a controller which is configured to receive said target image and to generate said output signal or said composite output signal. Alternatively, said electronic document reader may be connected to a second electronic device which generates said output signal or said composite output signal and sends it to the reader. The advantage of such an arrangement is that the second electronic device may have greater processing capability than the electronic document reader.
The invention further provides processor control code to implement the above-described methods, in particular on a data carrier such as a disk, CD- or DVD-ROM, programmed memory such as read-only memory (Firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and/or data) to implement embodiments of the invention may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (Trade Mark) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate such code and/or data may be distributed between a plurality of coupled components in communication with one another.
These and other aspects of the invention will now be further described by way of example only, with reference to the accompanying figures in which:
a and 1b show respectively, a front view and a rear view of an electronic document reader;
a shows a detailed vertical cross-section through a display portion of the reader of
b shows an example of a waveform for an electrophoretic display of the reader of
a is a schematic illustration of a typical colour electronic document to be displayed;
b is a flow chart illustrating one known method of processing the document of
c is a flow chart illustrating a method of processing the document of
d to 5g compare the results of sharpening on an image and text, respectively;
a to 7c illustrate a known technique for encoding a target image;
a to 8c illustrate a technique for encoding a target image according to another aspect of the invention;
d is a flow chart summarising the steps used in
a, 10a, 11a, 12a, 13a, 14a and 15a illustrate various target images on various backgrounds;
b, 10b, 11b, 12b, 13b, 14b and 15b illustrate the output to the driver to generate the target images;
c, 10c, 11c, 12c, 13c, 14c and 15c illustrate the real results of the outputs from
a and 16c show two sample images encoded using the method of
b and 16d show the two sample images of
a and 1b schematically illustrate an electronic document reading device 10 having a front display face 12 and a rear face 14. The display surface 12 is substantially flat to the edges of the device and may as illustrated lack a display bezel. However, it will be appreciated that the electronic (electrophoretic) display may not extend right to the edges of the display surface 12, and rigid control electronics may be incorporated around the edges of the electronic display.
Referring now to
A moisture barrier 102 is provided over the electronic display 104, for example of polyethylene and/or Aclar™, a fluoropolymer (polychlorotrifluoroethylene-PCTFE). A moisture barrier 110 is also preferably provided under substrate 108. Since this moisture barrier does not need to be transparent preferably moisture barrier 110 incorporates a metallic moisture barrier such as a layer of aluminium foil. This allows the moisture barrier to be thinner, hence enhancing overall flexibility. In preferred embodiments the device has a substantially transparent front panel 100, for example made of Perspex®, which acts as a structural member. A front panel is not necessary and sufficient physical stiffness could be provided, for example, by the substrate 108 optionally in combination with one or both of the moisture barriers 102, 110.
A colour filter 114 is optionally applied over the display. Such a filter is a mosaic of small filters placed over the pixel sensors to capture colour information and is explained in more detail below. The filter may be a RGBW (Red, Green, Blue, White) filter or another equivalent version.
Reflective displays, e.g. electrophoretic display media, are unlike most display technologies. When power is removed from conventional displays (such as LCD, OLED and Plasma) they revert to an off-state. This state is known and any colour can be driven accurately from this starting point. Reflective displays differ since they retain the last image that was written to them. Therefore, the display must be unwritten before it is rewritten. Waveforms are set of “transitions” that tell a pixel how to change from one image to the next; essentially a guide on how to turn every grey level to every other grey level. For a display capable of three grey levels this results in a waveform with nine transitions as shown schematically in
Referring now to
An external interface 1010 is provided for interfacing with a computer such as laptop, PDA, or mobile or ‘smart’ phone 1014 to receive document data and, optionally, to provide data such as user bookmark data. The interface 1010 may comprise a wired, for example USB, and/or wireless, for example Bluetooth™ interface and, optionally, an inductive connection to receive power. The latter feature enables embodiments of the device to entirely dispense with physical electrical connections and hence facilitates inter alia a simpler physical construction and improved device aesthetics as well as greater resistance to moisture. A rechargeable battery 1012 or other rechargeable power source is connected to interface 1010 for recharging, and provides a power supply to the control electronics and display.
Electronic documents to be displayed on the reader may come from a variety of sources, for example a laptop or desktop computer, a PDA (Personal Digital Assistant), a mobile phone (e.g. Smart Phones such as the Blackberry™), or other such devices. Using the wired (e.g. USB etc) or wireless (e.g. Bluetooth™) interfaces, the user can transfer such electronic documents to the document reader in a variety of ways, e.g. using synchronisation or “printing”. Electronic documents may comprise any number of formats including, but not limited to, PDF, Microsoft Word™, Bitmaps, JPG, TIFF and other known formats.
For transfer using synchronisation, the user connects the electronic document reader to a separate device (e.g. laptop or desktop computer, PDA or ‘smart’ phone) which is storing an electronic document. During this synchronisation, all of the electronic documents that are stored in any number of user-defined folders defined on the separate device, and that are not present in the memory of the reader are transferred to the reader. Similarly, any documents not present on the separate device that are present on the reader (for example, documents that have been modified or written to whilst displayed on the reader) may also be transferred back to the separate device. Alternatively, the connection interface may allow a user to specify that only a subset of the documents are to be synchronised. Alternatively, a live synchronisation may be performed, where the reader could store all documents that have been recently viewed on the separate device.
During synchronisation, the separate device takes control of the reader and transfers data to and from the reader. To understand the capabilities of the reader, the separate device may require several software components to be installed, for example, a printer driver; a reader driver (to manage the details of the communications protocol with the reader) and a controlling management application.
The incorporation of a printer driver or similar intermediary module to convert the electronic document into a suitable format for displaying on the reader allows transfer of the documents by “printing”. The intermediary module generates an image file of each page within a document being printed. These images may be compressed and stored in a native device format used by the electronic reader. These files are then transferred to the electronic reader device as part of a file synchronisation process.
One of the advantages of this “printing” technique is that it allows support for any document/file for which the operating system has a suitable intermediary module, such as a printer driver module, installed. During the file synchronisation sequence the control program looks at each document and determine whether the operating system associates an application with that file, for example, a spreadsheet application will be associated with a spreadsheet document. The control application invokes the associated application and asks it to ‘print’ the document to the printer module. The result will be a series of images in a format suitable for the electronic reader; each image corresponding to a page of the original document. These images will appear on the electronic reader, as if the document had been printed. The electronic reader may thus be termed a “paperless printer”.
An intermediary module comprising a management program 906 preferably runs as a background service, i.e. it is hidden from a general user. The intermediary module may reside in the document reader 904 or on the electronic device 900. The processing by the intermediary module may include adjusting or cropping margins, reformatting or repaginating text, converting picture elements within a document into a suitable displayable content, and other such processes as described below.
A graphical user interface 908 is provided, for example on a desktop of device 900, to allow a user to setup parameters of the paperless printing mechanism. A drag-and-drop interface may also be provided for a user so that when a user drags and drops a document onto an appropriate icon the management program provides a (transparent) paperless print function for the user. A monitoring system 910 may also be provided to monitor one or more directories for changes in documents 800 and on detection of a change informs the management program 906 which provides an updated document image. In this way the management program automatically “prints” documents (or at least a changed part of a document) to the electronic reader when a document changes.
The image information is stored on the electronic reader although it need not be displayed immediately.
a illustrates a typical electronic document to be displayed (e.g. printed) on the electronic reader. The document comprises different types of content, often described as objects, which are illustrated as separate layers for ease of understanding. The document comprises user interface elements 30 allowing a user to interact with the document, e.g. to select different menus. There are two different types of text content, black text 32 and white or other coloured text 34. There are also images 36, pixelated graphics with each pixel defining a specific colour (termed bitmaps) and mathematically defined shapes that are assigned with specific colours and thus form areas of block colour 38 (also termed vector graphics).
b illustrates how a colour electronic document is typically processed for display in black and white. At a first step S102, the electronic document is received in PDF, HTML or similar format. Such a format contains the text, image and vector graphics content. The document is converted in a rendering engine to a full colour bitmap (step S104). In a next step, the user interface elements are overlaid on the full colour bitmap (step S106) to create a final image which is in full colour. Other form elements and other scriptable pre-rendered content may also be added at this stage. This final full colour image is then sent to the display driver (step S108) which renders the image to black and white and optimises it for the display (step S110). The problem with this method is that there is typically little control over how the content is rendered to the display.
As explained in the background section, the process of printing coloured documents using a black and white printer often results in the loss of important information.
By rendering, it is meant, converting the document (or layer of the document) from its native format or code into an image suitable for output. Rendering may comprise first defining a bitmap and using that bitmap (and the unrendered image/bitmap) to determine the output. The output may be a waveform or set of waveforms which is provided to the display driver (i.e. to the active matrix driver circuitry). The waveform is a set of rules controlling the individual pixels within the matrix. For example, considering a simple case of changing between black and white, the set of rules comprises black to black, white to white, white to black and black to white. For a grayscale display having a variety of shades of grey, the set of rules is more numerous.
The first step is to receive the colour document and determine the different types of content S202. The dark text content may be rendered separately at step S206. Dark text may include dark grey, black or dark blue text. Accordingly, the first step of the rendering may include optimising the text colour, e.g. forcing all text of this type to black text. Where a colour filter is included, the text may be rendered at 150 ppi (pixels per inch) on a 75 ppi filter to improve resolution. The black text layer may be output as a fast waveform to make the text appear faster which may mean that it appears before other elements of the document. For example,
The white or other light coloured text content is rendered separately at step S204. As set out above, e-paper has only 16 colours whereas a full colour palette may have millions of colours. The intermediary module may store a look-up table which links the grayscale colours of the display to a predetermined number of colours from a full colour palette. The predetermined number of colours may be termed “native” colours. The rendering of the light colour text may include determining the colour of the text, determining which of the native colours is the closest match and setting the colour of the light colour text to this closest match colour. The light coloured text is preferably rendered separately from its background to avoid any dithering with the background.
The user interface elements are identified and rendered at step S206. The rendering may include determining the different types of user interface elements, e.g. text and highlights, and rendering each different type of user interface element separately. For example, the highlights (e.g. to show a user selection) may be rendered by determining the colour of the highlight and determining the best representation from the look-up table as described in relation to the coloured text above. The text may be rendered separately as described above and then overlaid. Additional image enhancement should not be required because the content has already been optimised by use of the other techniques. However, image enhancements, e.g. as described below, could also be used.
The rendering may also include using a novel waveform to create the illusion of animation by exploiting the fact that electrophoretic media is relatively slow compared to more conventional display technologies. The waveforms shown in
Possible Spatial Transition Waveforms include:
Customised “tags” either in XML or PDF or some other extensible mark-up language may be manually added to select the transition type. Alternatively, the transition type may be automatically selected based on content type.
Each image in the image layer may be rendered at step S208. The images may be processed separately or together. For example, standard techniques such as saturation boosting or sharpening may be applied independently to each image. For example,
The blocks of colour are rendered separately at step S212. In a similar manner to the rendering of the light coloured text, the rendering of the colour blocks may include determining the colour of the text, determining which of the native colours is the closest match and setting the colour of the light colour text to this closest match colour. The coloured blocks are preferably rendered separately from any text or other foreground to avoid any dithering with the foreground.
A final step (S214) is to combine the output from each layer to provide the overall waveform output. In practice the waveforms are more complicated than depicted in
One waveform may be used per page, but as set out above the ability to drive different types of content with different waveforms could be advantageous. A simple example would be to drive text with a very fast waveform and “fill in” the images with a slower more accurate waveform.
Once the area has been selected, a separate improvement algorithm may be run (step S310). For example, a look-up table may be provided to differentiate the plurality of colours which may be used in the colour image. The look-up table may be used to force the colour in the colour image to fit a best match colour. Alternatively, the look-up table may combine colours and patterns to provide a greater list of representations to differentiate the colours. For example, light blue may be represented by hash lines in the look-up table.
A final step (S312) is to combine the improvement to the specific area with the representation for the rest of the image and to output the overall waveform output representing the greyscale image.
As shown in
One disadvantage of using such a colour filter is that it effectively halves the true resolution. For monochrome (greyscale) content, the perceived resolution may be improved by rendering the monochrome content at “monochrome resolution” under the colour filter. The colour content is rendered at 75 ppi and merged with monochrome content at 150 ppi. This is reasonably effective for black and white text on a monochrome background but has little or no effect on coloured text, black or white text on a coloured background, coloured image or coloured graphics. Accordingly, an improved method is required.
The filter is controlled by using a mask which comprises a sub-mask for each colour of the filter, for example:
Out(i,j)=Rm(i,j)·I(i,j,R)+Gm(i,j)·I(i,j,G)+Bm(i,j)·I(i,j,B)+Wm(i,j)·I(i,j,W)
i,j are the co-ordinates in the rows and columns of the pixel matrix,
Rm(i,j), Gm(i,j), Bm(i,j), Wm(i,j) are the red, green, blue and white sub-masks, and
I(i,j,R), I(i,j,B), I(i,j,W) is the red channel, green channel, blue channel and white channel for the input image respectively.
The sub-masks are zero everywhere apart from where the appropriate colour is located.
a and 8a show the same target image (a red “P”). In
In
c and 8c render the results of
The method of
The method used in
Once the brightness information has been encoded at full resolution, step S406 turns to the colour encoding. For each bright (fully or partially) sub-pixel, it is determined whether or not the colour from that sub-pixel is required to give the target to create the output signal. For example, as shown in
The methodology of
In
When all sub-pixels for a pixel are on, for example, as with the pixels in the last columns of
In
In
In
a to 16d show real examples of the application of the methods of
No doubt many other effective alternatives will occur to the skilled person. It will be understood that the invention is not limited to the described embodiments and encompasses modifications apparent to those skilled in the art lying within the spirit and scope of the claims appended hereto.
| Number | Date | Country | Kind |
|---|---|---|---|
| 1209301.9 | May 2012 | GB | national |
| 1209309.2 | May 2012 | GB | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/GB2013/051346 | 5/22/2013 | WO | 00 |
