Patent Application
20170300925
The present invention relates to an intelligent label that is particularly constructed to be associated with a good, and to enable trusted and verifiable reporting of the condition of that good. In one aspect, the label's electronics is used from time to time to interrogate its display and verify that the desired message is actually perceptible, and in some cases may be useful for generating a historical record of what was displayed or perceptible as the product moved through distribution and use.
Modern commerce is increasingly dependent on transporting goods using carriers as society embraces more and more online shopping. For example, modern consumers are increasingly using online shopping and common carriers for delivering wine, prescription medication, food, and sensitive electronic devices. To assist in tracking and monitoring the movement of sensitive and expensive goods, labels have been developed in the past that incorporate RFID communication and intelligence. In this way, at the point of shipment and throughout the major carriers, the good has the ability to be tracked. However, adoption of such RFID labels has been slow, as the equipment for initializing, loading, updating, and interrogating the label's RFID electronics is expensive, and typically only available at larger transfer points in the shipping transaction. Further, it is unlikely, and even rare, for the end consumer to be able to interact with the label. Since the consumer is a critical part of the delivery chain, and the consumer is excluded from participation in the information available on the label, the use of intelligent labels has been quite low and very ineffective in improving the customer experience.
Intelligent labels, packaging, tags, windshield stickers, stand-alone displays and other devices, collectively referred to herein as “intelligent labels,” benefit from electro-optic devices that display messages that alert, update and inform the persons or machines proximate to them, as fully set forth in co-pending patent application Ser. No. 14/479,055, filed Sep. 5, 2014, and entitled “An Intelligent Label device and Method,” which is incorporated herein in its entirety. This earlier application describes an intelligent label that can be attached to any good, and then is used to provide a visual indicator to a human or machine on some condition or event in that distribution path. Of particular interest therefore are bistable and permanently irreversible electro-optic displays and intelligent labels that comprise them. In one example of using the intelligent label, if a good is subjected to an extreme temperature or to vibration shock, then a visual indicator may be set such that a human or a machine will understand that the good is no longer of acceptable commercial quality. Of course, it will be appreciated that machines can perceive information outside of the normal human optical range. Messages for the intelligent label are visually perceptible forms of data, information, content, text, patterns, images, shapes, symbols, codes, and colors, for example. It is important to note that these are visual systems and the messages may change one or more times over the life of the intelligent label. Further the power source that drives them may be limited or intermittent or susceptible to accidental or intentional disruption. Other components of the intelligent label may also fail or be subject to tampering. In this way, the message that is intended by the local electronics to be displayed on the intelligent label may not actually be what the user or machine perceives. Accordingly, in some applications the utility and value of intelligent labels may depend on the confidence (i.e. trust) with which the messages can be relied upon to make decisions and take actions, and further, that the actual messages perceptible at the time those decisions were made or could or should have been made, and actions were taken or could or should have been taken can be reliably and securely verified.
In one example, doctors and other healthcare professionals need to know that they will only be held accountable for decisions made and actions taken based on the information reliably available at the time. Hospitals need to know too. As do patients and insurers and everyone else with a stake in the outcome. And they also need to know that if something goes wrong, the system cannot be tampered with and of its information is trustworthy. In a specific example, a bag of blood has reached its maximum allowed time out of refrigeration, and the electronic circuitry on its attached intelligent label has instructed a message to be displayed on the bag's irreversible display that indicates that the bag of blood can no longer be used safely. In this way, the electronic messages stored within the intelligent label's processor and memory would indicate that at the correct time the visual indicator transitioned to show that the condition of the blood had changed was no longer safe. However, in some cases an electronic or logical failure may have occurred and the visual alert message was never perceptible to the nurse or doctor. To correctly assess liability for wrongly using the blood, it would be important to know precisely what was visible on the intelligent display at the time the alert should have been set.
In another example, an experimental drug may have its expiration date shortened due to a better understanding of the drug's deterioration over time. In such a case, an intelligent label may be updated remotely to remove the original expiration date, and replace the date with a new, shorter expiration date. A patient may wrongly continue to use the drug after the new expiration date, and may later claim that the new expiration date was never displayed. Accordingly, it would be important to know what, if any, change had been made on the label at the time the expiration date should have been changed. Having such a historical understanding of what was actually displayed could be critical to patient care and assigning liability. More particularly, in some cases what was actually displayed may not have communicated the intended message to the patient or care giver. For example, even if the intended message was correct, a defect in the display or display electronics may have caused an error in what was actually displayed, and therefore may have failed to communicate the intended message. More broadly, this problem occurs any time a manufacturer, distributor, or person in control of a product wants to update the information displayed to the user or consumer. Accordingly, it would be highly desirable that an intelligent label be able to determine if the intended message was perceptibly displayed at a particular time, and in some cases generate and maintain historical record of what was actually displayed for later evaluation.
It will be appreciated that different applications require different levels of confidence in the verifiability of the system. Often it is not enough to rely on an inexpensive processor having issued a command, an on/off button being switched, or a signal being sent. Particularly over time, excessive heat or cold, shock or vibration, humidity, disruption to power, component fatigue/failure, read/write/refresh errors, electrical interference, tampering etc. can all impact the integrity of a visual system and thus confidence in the ability to verify what was displayed, and further what was perceptible, at a given moment in time. In other words, it often is not enough to know what was supposed to be displayed or what may have been displayed at a different moment in time.
Conventional displays (CRTs, LEDs, most LCDs etc.) can be thought of as self-erasing. That is, such a conventional display is used to display an intended message to a user. However, there is no confirmation that the message has actually been presented in a way that is visually perceptible to the machine or human. For example, many internal and external factors can affect the visual perceptive ability of the display such as power disruption, excessive heat cold or humidity, shock vibration and pressures, and shorts and faults with the electronics or logic circuits. In some cases, feedback provided within these conventional displays may indicate that the intended message has been displayed, however these internal or external events may limit or distort what has actually been displayed to, and perceptible by, the outside world.
Messages displayed by bistable (or multi-stable state) displays such electrophoretic and certain cholesteric or nematic LCDs are to varying degrees stable without the continuous application of power. By design, they are however reversible and the displayed messages are therefore subject to accidental or intentional erasure or alteration. The displayed information therefore cannot be verified reliably visually. As used herein the term visual refers to messages, images and the like that are perceptible by both humans and machine. Certain messages however may be perceptible by machine but not perceptible by humans. For example, they may reflect light at wavelengths outside the human perceptible range of (generally considered to be 770 to 390 nanometers). This characteristic may be exploited in several ways, for example to create watermarks and other messages in response to events that are machine perceptible (readable) but not perceptible by humans. The verification systems and means described herein may be utilized with such non-visual, but machine perceptible messages.
Accordingly, there is a need to reliably verify that an intended message has been presented on a display in a visually perceptive manner. In some cases, the stakeholders would also benefit from generating and maintaining a historical record of what was actually perceptibly displayed on the label.
An intended message is written to a display, which may be bistable. Pixels that comprise portions of the message are measured and evaluated to determine if the message actually displayed on the display was perceptible by a human or a machine. In some cases, information regarding the message actually displayed on the display may be stored for later use, irrespective of whether or not the display provided a perceptible message. Responsive to determining that a message is perceivable or not perceivable, alarms may be set, one or more third parties notified, or additional display features may be set.
In one example, the perceptibility of a message written to an intelligent label can be verified. More specifically, electronic circuitry within the intelligent label writes an intended message to a bistable display. Electrical characteristics of the pixels on the bistable display are measured, and a contrast and color profile may be generated. This profile represents the actual message that would have been perceivable by a human or a machine. This actual message can then be compared to the intended message, and a level of confidence that the proper message was presented can be generated. In this way, it can be verified that the proper message would have been perceivable by a user or human at a particular time, for example, when a severe environmental event occurred. Further, a historical record may be generated and maintained regarding the visual state of the bistable indicator at various times in the lifecycle for the product during use and distribution.
Advantageously, the verification process and label disclosed herein allows the visual information on the label to be confirmed and it verified that a proper message was displayed and perceptible at a particular point in time. And further, independent of the intended message, the verification processes and labels allow for determination of the actual information displayed, its meaning and its perceptibility. In this way, liability can be more accurately assessed, and the trustworthiness of the entire distribution and use cycle is dramatically improved.
The intelligent label may take many forms, such as a traditional style label for attachment to a discrete box or package, it may be integrally formed on a package such as a shipping container or mailer, or it may take the faun of documentation that accompanies a shipped product. In other examples, the label may be integrated or applied on prepaid gift cards for example, or can be integrated into the good itself. Generally, the intelligent label is intended to enable a highly trusted, robust, and accurate way for safely and securely confirming or reporting a change in the condition of a good, for example, while a good is transported from a point of origin to a consumer or it is held in stock prior to use. Additionally, the intelligent label enables analytics and an understanding of the quality and handling of the good over time that is not available with prior systems. Further, the intelligent label provides accurate and timely information to various participants in the handling and use process, including the end user, without the need for sophisticated processing, communication, or interrogation systems. In this regard, the intelligent label has a simple electro-optical display (indicator) for visually presenting selected important information about the quality and handling of the good. A preferred option is an irreversible bistable indicator that cannot be turned back to its original state electronically, and thus is naturally resistant to tampering or accidental alteration. In some labels one may use both bistable and irreversible indicators corresponding to different indicator functions. It will be appreciated that in some constructions the electro-optic indicator can be constructed with electrochromatic material or electrophoretic material.
Bistable indicators may be used to temporarily present a code or information. Bistability means that the display or the indicator changes from a first optical state to a second optical state by using a powering protocol, and remains in the second optical state without the application of additional power. However, this state (the second optical state) can be reversed to the first optical state by applying a different powering protocol and can also be maintained in that state without subsequent application of the power. The length of stability in a given optical state is dependent on the application requirement and a suitable electro-optical display/indicating system meeting that requirement can selected. In some applications optical state stability without the application of power, on the order of a few minutes may be acceptable, while in other cases this may extend to several days, months or years. Certain non-emissive electro-optical systems such as electrophoretic, liquid crystal and electrochromic systems can be tailored for various bistability requirements. Another desirable property of these indicators is their environmental durability (time, temperature, humidity (moisture), pressure and radiation (e.g., UV) in both activated and non-activated states so that it is obvious from visually observing the indicator its last state of activation (or inactivation). This environmental stability ensures that it would be difficult to mistake the conveyance of its intended optical state and also difficult to tamper with and also results in a permanence of indicated information.
Referring now to
Intelligent label 10 typically has an actuator, not illustrated from the front, that activates the electronics in the label just prior to the label being attached to the good. For example, the label may have adhesive backing, that when removed, enables the capture of the particular date and time when the label is being attached to the good. To enable this, the processor would operate in a very low power state to maintain its timer, and then when the adhesive is removed from the back of the label, a higher power mode would be enabled that allowed capture and storage of the current time and day. In this way, the label itself can accurately capture when it is attached to a good in-service. The good may then be placed into the shipping chain, where at each transfer information may be captured from the label using the barcode 12 or from the electronic RFID communications, and additional information may be stored in the RFID areas as well, provided such RFID equipment is available at shipping locations. Actuation, or initiation of the electronics also may be via a variety of means including a remote signal (e.g. RF, optical or electrical) a mechanical, electro-mechanical or electrical switch.
Intelligent label 10 also has an electro-optic display area 13 for providing immediate visual information regarding the quality of the product without the need for interrogating the RFID communication system. In one example, the processor in the intelligent label 10 contains rules as to how long the shipping process should take. In a more specific example, the label 10 could be applied to a box of flowers. The shipper-grower may require that the flowers be delivered within a two-day time span. As soon as the label is applied to the flowers, the timer starts and begins counting the elapsed time that the flowers have been in the shipping process. Initially, the intelligent product label may indicate the flowers as being fresh, but if the shipping time goes behind the limits set in the rules, the processor may cause the electro-optic device to indicate not to accept the flowers. Thus, at any point in the shipping process the receiver would be put on visual notification not to accept the flowers. This point could be at the end consumer point, or could be at any other point in the shipping chain. In one interesting alternative, there could be a period of time when the product is not as fresh as implicit in the buyer's initial order, but yet would be acceptable by most consumers, particularly at a reduced price. In this case, the intelligent label could be set up to inform the end-user to call customer service. Upon calling customer service, the customer may be offered a discount or other incentive to accept the flowers, acknowledging that they are nearing the end of their freshness state.
In a more specific example of the electro-optic display, the electrochromic display may have an informational block 15 for providing additional specific information. The information in informational block 15 may provide coded information dependent upon specific attributes of the shipping process e.g. elapsed time, monitored conditions etc. Typically, the informational block 15 would be activated in order to give more specific information as to the broad information given in area 13. For example, the intelligent label 10 shows that the product associated with this label should not be accepted. If, for example, a consumer removes a package from their mailbox with the “do not accept” highlighted, the consumer typically will not have the equipment necessary to interrogate the label through its RFID communication channels. However, at the time the “do not accept” electrochromic indicator was set, the label also provided an electrochromic indication that provided the additional information as shown in information block 15. Accordingly, when the consumer calls customer support for the provider of the good, the consumer can visually read the code included in block 15 to the customer service representative, and that particular code can be associated with a specific time or event causing the good to go bad. In this way, customer service obtains significant information that is accurate and trustworthy as to where in the shipping chain the product was mishandled. By providing such information, the chances for fraud are decreased, and the opportunity for improved customer service is enabled.
Label Construction
It will be appreciated that the intelligent label may take many forms, but for convenience, the structure and function of the intelligent label will be described with reference to a discrete label having an adhesive backing for attachment to a mailing package or good. It will be understood that other constructions for the intelligent label are consistent with this disclosure, such as a label integrated with a package, integrated onto shipping packaging, or integrated into shipping documentation. It will also be appreciated that other constructions or possible consistent with this disclosure.
Print Information. Referring now to
Processor. Intelligent label 25 also has electronics on or embedded within its structure. Electronics includes a processor 28 having an associated clock 29 and storage 30. The processor also manages communication using communication processor 34, which typically is an RFID radio. It will be appreciated that the various electronic components may be implemented using a single integrated circuit device, or may require multiple devices.
Power. The electronics for intelligent label 25 require a power source 32 for operation, communication, and transitioning the electro-optic message indicator 33. This power may be supplied, for example, by a traditional primary or a secondary cell battery, a set of thin-film layers constructed as a battery, capacitor, or may be an antenna structure constructed to generate power responsive to an RFID field signal.
Message Indicator. Also on the front side of the label, there will be an electro-optic message indicator 33, which in one construction may be, electrochromic or electrophoretic indicator, for providing additional information regarding the condition or quality of the good. The message indicator 33 may be bi-stable or permanent, i.e., irreversible. More particularly, the electro-optic material may be specifically formulated and activated in a way that it has two color or transparency states, with the electro-optic material remaining in the first state until it is activated to transition to the second state. Once electrically transitioned to the second state by the processor, this process is irreversible, and the message device 33 remains permanently in the second color or the second transparency state. The particular formulation of the electro-optic material is fully set forth in published U.S. patent application number 20110096388, which is incorporated herein in its entirety.
In one example, message indicator 33 may be a first color while in the first state, and then when transitioned to the second state, visually present a second color. In another example, the electro-optic message indicator may change its transparency state. In this way, electro-optic message indicator 33 could be placed over printed information that would not be visible while the message indicator is in the first state, but then when transitioned, information below could be viewed through the now transparent message indicator. In another example, the message indicator 33 may be more complex and structured in a way that can build textual or numeric information, for example, such as using a segment or dot-based character construction. Further, although the message indicator 33 is described as having only two states, it will be appreciated that some indicators may have more than two stable states. In another variation, upon activation, the message indicator may transition from a first state to a second state irreversibly, and then upon further activation transition reversibly between the second and a third state, i.e., show bistability between the second and the third states.
In operation, electro-optic message indicator 33 would be transitioned according to rules set in the processor for the particular good that is being shipped. These rules would be implemented using processor 28, and when a rule is satisfied, the processor 28 would cause an appropriate electronic signal to transition the electro-optical material in the message indicator 33. For example, rules may be set that would cause the message indicator 33 to indicate whether or not the package was shipped and delivered within the allotted time.
In some cases, the electro-optic message indicator 33 may be structured as bistable electrophoretic indicator. Electrophoretic technology is most typically seen as an electronic paper display useful for handheld readers and small electronic displays. Electrophoretic technology is well known, but generally, an electrophoretic display forms images by rearranging charged pigment particles (pixels) with an applied electric field. Upon the application of an electrical signal, the pigments reorient themselves so that either a white side or a colored side is presented to the reader. By changing the electronic signal, the pigment can be oriented to the other visual state. Electronic paper, e-paper and electronic ink are display technologies that mimic the appearance of ink on paper. Unlike conventional backlit flat panel displays that emit light, electronic paper displays reflect light like paper. Many electronic paper technologies hold static text and images without electricity, but the display can be modified and changed upon a new application of power, or upon electronic or logic error. Although the intelligent labels are described generally using an electrochromic display, it will be understood that an electrophoretic display or a more general electro-optic display will work.
For example, it may be useful in some applications to use an electrochromic LCD display, that is, a display that is not typically considered to be either bi-stable or permanent. Even with such a display, it would be desirable to determine it, a particular time, the correct intended message was being displayed in a perceptible manner, and if not, to capture the pattern that was actually displayed. Herein, pattern is used broadly to include any arrangement of pixels, whether or not the pattern of pixels creates a human-recognizable symbol or character.
Communication. Bidirectional communication may be provided with intelligent label 25 using the communication processor 34. The communication processor 34 may be an RFID radio, although other radios such as ZigBee, 802.11, or Bluetooth maybe used. The communication processes on communication block 34 may be controlled by processor 28, with processor 28 managing what information is sent and received through the radio. Once information is received, it may be stored in storage 30, or rules may be applied to determine if action needs to be taken, such as setting the electro-optic message indicator 33.
Actuator. Intelligent label 25 also has an actuator 31 for determining when the label is being attached to a good, for example. In this way, the actuator provides an accurate indicator of when the good is entering the shipping chain. The actuator in some cases may be provided as a physical mechanical device, and in other cases may be optical, electrical, electro-optical, RF, or chemical. It will be appreciated that the electronics man be activated at other trusted and confimiable times depending on the specific application. Actuator 31 can take many forms, depending upon the physical structure of the intelligent label 25. In one example, the actuator 31 is constructed along with the backing of the label, such as when the backing is removed to expose adhesive, the processor is provided with a signal that the label is about to be attached to the good. In this way, the processor can then store the accurate information regarding how the product entered the shipping chain, which can provide useful and accurate comparison information throughout the shipping process. In practice, the actuator can be implemented in many alternative ways. For example, the actuator may be set such that the removal of the label backing breaks an electronic circuit that can be detected by the processor. In another example, the removing of the backing material and placement of the label on the good may close a circuit, thereby giving a signal to the processor that the label has been attached to the good. In other examples, the actuator may be pressure activated through the application process, or provide an electronic signal that is generated by some physical action, such as by pulling a tab. It will be appreciated that actuator 31 maybe implement it in a wide variety of ways.
Sensor. Referring now to
The processor within intelligent label 50 will have a set of associated rules for the expected environmental conditions that the sensor 55 should be exposed to these rules can be set to simplistically monitor the sensor data, or may contain more sophisticated algorithms as to allowable conditions. For example, a good may remain in a quality state if exposed to a temperature for a short period of time, but would be considered out of specification if the temperature remained for more than a set period of time. It will be appreciated that a wide variety of rules may be set for sensor 55.
With the addition of one or more sensors, it is likely that the intelligent label 50 will need at least one more electro-optical message indicators 53. It will be appreciated that several electro-optical indicators and even of different types may be useful depending upon the number of sensors and sophistication of the rules for the good associated with intelligent label 50. In one example, an electro-optic indicator may be provided for visually indicating the letter, character, or code that provides more information regarding when or why the good was deemed to be unacceptable. Again, as the electro-optical indicator provides a human readable visual indicator, a person, such as the end consumer, would not need to wirelessly interact with the intelligent label 50 to obtain meaningful information regarding the state transition. In this way, a customer service representative interacting with the consumer would be able to not only verify that the consumer's product has been indicated to be a bad quality, it may be able to determine additional specific information that could improve the shipping process, and provide valuable information in satisfying the customer's needs.
State Verification. In some cases, particularly with high value goods, it may be desirable to add another layer of confidence that the electro-optic message indicator has properly transitioned to its second state. Referring now to
Referring now to
Message Verification
Pixels, as discussed herein, will be understood to be single addressable visual elements of the display. In some instances, a pixel may be a ‘dot’ and in others it maybe a shape such as a ‘segment’ used in the formation of a ‘seven-segment’ alphanumeric display. Pixels may also be a variety of shapes, symbols or images that are determined by the surface areas of the electrodes used to signal them. A shape of course may be comprised of multiple pixels. In many applications such as intelligent labels, the density, variety and resolution of the displayed messages is not typical of that required for consumer electronics. As such the messages may be generated using comparatively large pixels in shapes optimized for messages appropriate for the application instead of arrays of much larger numbers of significantly smaller pixels. Accordingly, a message consists of the visual ‘state’ of one or more pixels. In a monochrome display for example, a pixel typically has at least two intended states, on each of two high contrast colors (e.g. black and white) and depending on the display, a third (or more) state which is not one of the high contrast colors (e.g. gray or semi-transparent), but sits between the two high contrast colors. The perceptibility of a message or visual information further typically depends on the actual visual state of the message pixels and that of the pixels adjacent or proximate to them.
In certain electro-optic displays the visual state of a pixel corresponds to an electrically measurable characteristic of the pixel (e.g. voltage, resistance, capacitance, etc.). U.S. Pat. No. 6,995,550 B2 “Method And Apparatus For Determining Properties Of An Electrophoretic Display” and referenced patents and applications describe such methods for electrophoretic displays. A pixel thus has a visual state and a corresponding electrical state. A pixel also may be considered to have a single ‘state’ that has both a visual and a corresponding electrical characteristic. The location or ‘address’ of each pixel relates to the electrodes used to set its state.
Described herein are systems and means for verifying that the messages actually displayed by bistable electro-optic displays are the same as those intended. That intended message may be determined for example by the output of the intelligent label's driver circuitry, by instructions/data in the intelligent label's processor/memory, or by monitoring signals transmitted to or received by the intelligent label. At a later time, this intended message can then be compared to the message that was actually perceptible on the display.
Verification of Perceptibility
Referring now to
Block 76 shows that the processor or driver circuitry attempts to write the intended message to the electro-optical display. In some cases, this intended message may be stored for later use in comparing to what was actually perceptibly displayed. It will also be understood that the intended message may be interrogated in various areas of the electronics, such as by evaluating contents of memory locations or evaluating driver circuitry. It will also be understood that method 75 may be used multiple times during the life of an intelligent label, since the intelligent label may have multiple messages that may be definable at various times in the distribution cycle. Method 75 may also be used to determine or confirm the continued perceptibility of the same message. Generally, the message comprises a set of pixels that when viewed together present the intended message. Depending upon the particular type of display technology used, these pixels may be white, black, gray, or set to another color. Typically, the pixels on the display will be activated using an electrode or electrode set to provide an electric stimulation or signal to each individual pixel. It will be understood that the process for activating pixels with electrodes is well understood.
At this point of method 75, the processor or electronic control circuitry has made an attempt to set an intended message onto the display. However, there is no feedback or verification that the intended message actually has been displayed. Accordingly, block 78 shows that the state of the pixels may be electronically measured to determine their visual state and the perceptibility of any message or information. That is, the individual pixels may be interrogated to determine if they are black, white, a particular color, or some state in between. Often, the interrogation of the pixels may be done using the same electrodes used to set the pixel, however in other cases other separate detection electrodes may be used. Also, it may not be necessary to evaluate or measure all of the message pixels. For example, there may only be a subset of the pixels that are required to perceptibly present the intended message. However, in many cases the simplest process is to use the same electrode used to set each pixel, measure the electrical characteristics of every pixel of the intended message, and store the measured electrical characteristics. It also may be advantageous to measure the electrical characteristics of the pixels adjacent to the message pixels or of reference pixels.
Each of the pixels can now be evaluated to determine if they are of the intended color and contrast. That is, it can be determined if the intended black pixel is actually black, or if it failed to transition at all, or if it is some level of gray in between. In this way, it can be understood if each pixel is properly set, and if not properly set, how close to the correct state it is. Accordingly, as shown in block 79, the intended pixel states corresponding to the intended message can be compared to the actual pixel states of the display.
In this way, it may be determined if the message visually perceptible to a user or machine is the intended message. For example, if all the pixels states are as intended, then the perceptible message is the intended message. If a few pixels are not fully set to their intended state, it may be determined that although not perfect, the intended message is still perceptible. In other cases, the differences between the intended states and the actual states may be so significant that it is determined that the message being displayed would not be perceptible by a user or machine. In making this determination, it may be useful to determine a subset of pixels that are actually necessary for generating a perceptible message. For example, a few pixels forming the base of a symbol may not have been properly set to a dark contrast, but having those few pixels properly operate is not necessary for a user to perceive and understand an intended message. That is, whether or not those pixels are black, white, or gray, their particular state creates little or no ambiguity in the perceptibility and understanding of the message.
With a level of confidence in the perceptibility determined, the perceptible information can be stored for later use, may be transmitted to a third-party, may set an alarm, or maybe used to set another new local message. For example, if method 75 determines that a new expiration date is not actually perceptible, the intelligent label may locally cause a second display to present a large red dot showing that there is a severe problem with quality. In a similar way, the label may cause an alarm or message to be sent to a third party, and an indication of the perceptibility of the expiration date as well as that of the red dot, may be stored for later use in verifying what message or information was actually available to user at a specific time. As illustrated by the arrow from block 81 to block 78, this verification process can be used multiple times to determine what was actually displayed at various times throughout the distribution and use cycle of the product. Finally, as shown in block 82, it can be determined if the proper message or messages have been perceptibly displayed for use by a human or machine throughout the product distribution and use cycle. In one application, the intelligent label has an indicator for informing a viewer that the message has an ambiguity. In this way, the user can take extra care in evaluating the message.
In applying method 75, it will be understood that algorithmic comparisons can compensate, adjust and account for errors in the measured results. Error correction techniques may also be applied. Confidence values/indexes may be generated/employed using the measured values, the importance of particular pixels to the perceptibility of the information, the accuracy of the perceptible information (and the benefits and risks of actions taken, or not, accordingly). In some instances, the comparison of measurements corresponding to the intended and measured information will be advantageously conducted off the label at the network level (e.g. to enable 3rd party verification/auditing).
In one example of method 75, a bistable display uses a seven-segment display technology to present an alphanumeric character to a user or machine. The intended message, for example the number “7” is stored in memory. Three pixels (segments) out of the seven that characterize an alphanumeric character are ‘addressed’ by the appropriate electrodes (the top segment and the two vertical segments on the right). Signals are sent to color the three pixels with a first color (e.g. black). Independent of their last-known state the signals are preferably sent to each of the four other pixels comprising the seven-segment character to color them with a second color (e.g. white). Depending on their current state this may involve refreshing (re-writing) certain pixels' existing states. Optionally, signals may be sent only to pixels required to change state as determined by their last-known state or last-known-good state. If the state of the pixels proximate/adjacent (surrounding or interstitial to) to the seven pixels comprising the character can be electrically changed (e.g. the pixels are addressable) these pixels may also be sent signals to maximize the contrast between them and the message pixels and thus the perceptibility of the displayed information.
Using the same electrodes used to set the pixel's state, at least one electrical characteristic of each of pixel in the pixel-set appropriate to the intended message is measured. The pixel set in this example is preferably the seven pixels that make up the seven-segment display and the surrounding pixels. In another example, only the three segments that needed to be transitioned might be measured. However, greater confidence would be established if all seven segments were measured. But, in some power and time critical situations it may be advantageous to measure fewer than all the pixels.
The results of the electrical measurements are algorithmically compared to the previously stored intended message using predetermined measurable values corresponding to the different visual states possible for each pixel. The comparative results are stored in memory along with date/time and other device and event information as appropriate. If the result of the comparison is not as intended an error message/alarm is generated and wirelessly transmitted and optionally visually displayed (which itself might be subject to verification).
Referring now to
Referring now to
As described above, it may be important to determine the visual content perceptible at a specific moment in time independently from its relationship to an intended message. The perceptible content may consist of a message that is different from the intended message, incorrect or misleading yet importantly it might be sensible given the circumstances. Or it might be completely devoid of meaning. The term content should be construed broadly to encompass whatever is actually displayed independent of its coherence or meaning.
Clinical research organizations and healthcare professionals for example, need to know that their decision to dispense a drug or infuse a patient with blood can and will be evaluated on the basis of the information available to them when the decision was made (not before or after). It may not be enough for an intelligent label to know that the information displayed wasn't correct. A user may not be able to tell the difference (it may ‘look’ correct/reasonable). For example, the difference between an expiration date, temperature reading or access code that ends in a zero or an eight or a four or a nine is a single pixel (segment). For better or worse a user may make a decision and take an action based on the immediately available information that may be erroneous. A single pixel (segment) can also be the difference between a plus and a minus sign or the international OK/Go and Warning/Stop symbols.
Measurements of the electrical state of the pixels in a pixel-set can be used in conjunction with appropriate reference tables or character recognition technique/systems to determine the content displayed and the messages it contains, if any. Measurements of the electrical state of the pixels in a pixel-set can be used to reconstruct the content actually displayed. That reconstruction in turn can be viewed, interpreted, analyzed, evaluated etc. by a person (or persons) or machine. And in conjunction with associated time/date information, an accurate, if not exact, reconstruction of the content displayed at a past moment in time could be used to judge whether or not a message was sufficiently perceptible and understandable to support a decision or action.
Measurements of the pixels' electrical characteristics can be taken at any time: before and immediately after updates, periodically over time, or in response to a command. Such measurements can be used to compare the original visual state with one or more past ones or absolute references to determine how they've changed over time or when the contrast in absolute terms or relative to each other pixels has unacceptably diminished. In the latter case the intelligent label could refresh the pixels or issue an alert.
These measurements can be taken and written to memory with numerical precision consistent with the accuracy of the measurement means and the pixel configuration (e.g. larger pixels are easier to measure), the number of pixels and number of measurements, and the power and memory available. The measurements can also be translated into a set of variables consistent with their visual states e.g. black, white or neither black nor white (e.g. gray, indeterminate, transparent, semi-transparent or opaque). These measurements can be compared over time to determine the stability/efficacy of the display and perceptible information (e.g. to environmental conditions, time). In this way, this process can be used to determine if the display in an intelligent label or other electronic device (e.g. consumer e-reader, tablet or phone, electronic shelf label or commercial display in a public venue) is fading or otherwise failing (or damaged, tampered with etc.) and can be used to predict failure or the need to refresh the appropriate pixels or take other action such as generating a warning or error message. As with the verification systems and means previously described, the systems and means used here can be to varying degrees local to the intelligent label or to a remote display.
In some instances, it is advantageous for an entity independent of the system owner or operator to be able to verify the presence, accuracy and perceptibility of messages at various moments in time, location or other conditions, especially when consumers are involved. For example, owner/operators of an electronic shelf label (ESL) system (or electronically updatable price tags) retailers (or their operators) have the ability to update prices and other messages consumers see when making purchase decisions. Retailers also have the ability to control the prices the consumer pays at check-out/the point-of-sale. At worst this this creates an opportunity for abuse/fraud. At best it creates uncertainty and doubt. Consumers, regulators and honest businesses have an interest in being able to simply and reliably verify the accuracy and consistency of the pricing messages actually used to make purchase decisions and payments.
This can be achieved by sending signals to the appropriate pixels in an ESL required to display the current price, immediately measuring the electrical characteristics of the appropriate pixels, and then algorithmically converting the measured results into displayed information (the displayed price) and storing the results along with the date/time and information that identifies the ESL or the SKU or other item identifier. Further, the process controls must be such that the measurements and storage of the displayed information (displayed price), date/time, ID etc. occurs every time signals are sent to alter the pixels. And preferably, new signals can't be sent until the displayed information is determined and stored.
The required process control can be accomplished in a number of ways. One way is to separate or isolating microprocessor functions such that the retailer for example can update the display, but cannot interfere with automatic measurement of the pixel's electrical characteristics immediately thereafter. Nor the storage of the results and the algorithmic conversion into displayed information. Nor would they be able to access that information or associate date/time ESL ID or product SKU etc.
As stated elsewhere the functions and process controls can be distributed advantageously. The microprocessor functions could be isolated, yet securely communicatively coupled as required within a single IC or split between two ICs. Control can partly be manifest over the driver circuitry because the update and monitoring functions would both preferably use the same electrodes.
The price charged for the item (SKU) at the POS and the time it was charged are typically printed on a receipt and electronically recorded/stored (with appropriate security, and preferably with a third party). Both the visual and electronic record (with proper security) can be electronically compared to the price displayed at the time (or most recently). This provides a secure framework for auditing pricing information or a 3rd party to act as a neutral monitor/trustee in regards to pricing (and other information).
In some implementations the intended messages would be advantageously displayed as composite images consisting of multiple pixels. A symbol (e.g. a ‘dot’) or icon, letter or number, for example could be comprised of multiple smaller pixels in the appropriate state (color) versus a single large pixel. To limit the complexity of the measurements, computations and conserve available power, an acceptable level of verification of the displayed messages may be achieved by determining the pixel-set that makes up the intended message (or a component of the intended message), sending appropriate signals to the pixels in the pixel-set to produce the electrical state that corresponds to the desired message, and then measuring the aggregate electrical state of a set of pixels that comprise the intended message. That pixel set could be, for example, either all of the pixels that comprise the intended message, or in some cases a subset of the pixels that comprise the intended message. The process then algorithmically compares the measured results of the pixel-set with the corresponding previously determined pixels that comprise the intended message. It would be desirable that the algorithm compensates for ‘noisy’ data to compensate for activation (write) errors and measurement (read) errors.
Depending on the precision of the measurements, one or more pixels of the set of pixels that make up a large visual element e.g. a symbol, image (including a predetermined alphanumeric text string) may be measured and aggregated and used to represent the state of the visual element. Pixels may be sized according to the requirements (e.g. min size/shape)
The perceptibility of displayed messages, information, images and the like is directly related to the contrast between the pixels that make up the message. In other words, the ability to perceive a message depends on the contrast between a first, visual or colored state (e.g. white, or less than white) and a second, colored state (either black, or less than black) of the pixels. Note that the perceptibility of a message is not dependent on both, or either color being ‘pure’ (e.g. 100% black or 100% white) and the differential being 100% (it will always be somewhat less). What matters in most cases is that the differential in contrast (or contrast ratio) between the two states is sufficient for the message to be perceptible—not the absolute states in and of themselves.
A variety of algorithmic relationships may be utilized, but by way of illustration only, a display with an 80% ‘white’ first colored state can be thought of as also having a 20% black first colored state, and an 80% black second colored state may be thought of as having a 20% white second colored state. The differential in contrast between the two states would be 60%. The same would be true for a display with a 90% white first colored state and a 30% black second colored state.
If a 60% differential in contrast is sufficient for a message to be perceptible, then a message would be perceptible on either displays (80/20 and 90/30). A lower differential in contrast, e.g. 50% would indicate that perceptibility of the message was limited or impaired. The relative contrast of the two states is associated with a measureable characteristic. Thus a “contrast differential” can be used to verify or determine the visual state of a pixel or pixel-set and by extension the perceptibility a message. Note that optical states other than contrast, such as color, may also be used separately or in combination.
Another way of looking at contrast is to consider a contrast ratio. The human eye adjusts to the (average) light intensity in the image, by changing its pupil diameter (fairly quickly) and also light sensitivity of the retina (over several minutes). This is similar to a linear gain change in an (otherwise) linear photo detector. Simplistically, the eye sets the max intensity to the highest level of the pixels in the display (in practice average would be a better measure). Thus for the two cases: 80/20 would be normalized to 100/25, and 90/30 would be normalized to 100/33. It would therefore be easier for the eye to see the 100/25 (with a contrast ratio of 4:1), than the 100/33 (with a contrast ration of 3:1).
Note that if the charged/colored particles in electrophoretic displays are in other than their intended or “set” (bistable) state (either because they migrated from their set position or they're weren't adequately set to begin with) then they will be interspersed with other particles. This will have the effect of reducing both the optical contrast differential and the measurable electrical differential, relative to reference values.
A “contrast differential” as the term is used herein, is the difference between a first and second measurement of an electrical property of a pixel or pixel-set that corresponds to a first, visual (“colored”) state and a second, colored state. The “contrast” of the colored states (perceptibility correlated to them, contrast differentials) being determined relative to each other subjectively (e.g. by human trials) and/or objectively (e.g. by measurement of light reflection or absorption). Such determination can be made for each individual display (e.g. factory calibrated) or for each class of device, or other criteria.
In an example method all the pixels comprising the display are set to a first state where the pixels are colored to a first color (preferably the background color which is preferably white) and then a first measurement of an electrical characteristic is taken. The pixel or pixel-set that comprises the desired message is set to a second state where the pixel or pixel-set is colored to a second color that contrasts with the first color and a second measurement of the electrical characteristic is taken. Finally, the contrast differential is calculated between the two measurements. The calculated contrast differential then may be algorithmically compared to one or more previously generated reference “contrast differential,” values, indexes or benchmarks).
Preferably, immediately prior to the first measurement, the pixel, pixel-set or all of the pixels comprising the display, is set to a first color, the first color being the background color (e.g. white). In the above example, all the pixels in the display were set to a first state, color and especially the background color (white). This has the effect of maximizing the contrast between the message pixel-set (the ones comprising the message and is this example, black) and the surrounding pixels (by setting them to white). A subset of all the pixels in the display, those that were known to surround the pixel set, could be used instead (and they could be set pre or post the setting of the pixel-set to the send color (black). And although less reliable, only the pixels surrounding the pixel-set that were last known (measured) to be of set of the second color could be used set to the first color, pre or post the coloring of the pixel-set.
In some cases, the contrast differential is determined without setting or resetting the states of the pixels in the pixel-set. For example, to compare it to a previous contrast differential and determine if the message has been stable over the interim interval or the display's efficacy impaired. The first color may be either dark (e.g. black) or light (e.g. white) and can be of colors other than black or white. The relationship between the electrical characteristic and the perceptibility of the message may be linear or nonlinear. Previously generated reference contrast differentials may be generated during the manufacturing process and supplemented by an optical verification system. Or, they may be generated at one or more times during the display's (and intelligent label's) life. As previously described, conditional rules/logic may be applied and actions taken in response to comparisons between contrast differentials.
As noted previously, the relationship between the optical state of a pixel and the corresponding electrically measured characteristic is required (e.g. for a given voltage or capacitance, there is a corresponding optical state). This information may be obtained by calibrating the electro-optic displays using automated means consisting of electrical measurements of the electrical state of pixels of different sizes, surface areas, shapes, volumes, thicknesses, masses, manufacturing defects and tolerances, etc. and optical measures of the displayed information. Further, the information may be calibrated to compensate for different environmental conditions (e.g. temperature or humidity).
In certain instances, it may be advantageous for an intelligent label to create a verifiable visual alert/alarm based on the displayed message. And further, that the visual alert/alarm is, unlike that produced by a reversible bistable electro-optic device, irreversible permanent or near permanent (like a photograph). For example, if the displayed message was not as intended, a red dot or other symbol or message could be displayed to visually, and verifiably alert the user that the message displayed by the bistable electro-optic device was either incorrect or potentially inaccurate. Of interest, therefore is an intelligent label comprising a bistable display and a permanently irreversible display. In a preferred embodiment, the intelligent label comprises an integrated electro-optic display comprising a bistable electrophoretic layer and a permanently irreversible electrochromic layer (such as that described in patent application US20110096388A1 “Flexible and Printable Electro-optic Devices) and common electrodes. And further, that the reversible electrophoretic and irreversible electrochromic layer utilize the same electrodes layers.
Given the relatively small values for the measured electrical characteristics of electrophoretic pixels, it may be advantageous to include in the display or intelligent label means to minimize or compensate for electrical noise in the measurement system. Electrical noise is inherent in any electronic system design and is normally a strong design consideration. Electrical noise is generated both internally from switching digital logic, as well as from external sources, e.g. nearby electrical components, RF sources, etc. that are coupled into the system through internal wiring or signal traces, antennas, etc. Noise may be mitigated through design techniques, e.g. ground planes, layout and filtering so that it is typically well below those of the activation signals.
In the case of electrophoretic displays the noise is typically well below that of the signals that switch the display elements. It may however be on par with electrical measurements of pixels intended to determine their state. In such cases, in addition to executing good ground plane and signal routing designs, the display and associated electronics may take advantage of common-mode techniques (E.g. common-mode noise rejection techniques) so that the measurements “ride” on top of the noise to the extent feasible. Additionally, they may use averaging of multiple measurements and comparisons to average reference levels to determine a difference between measured and reference levels. Other signal processing techniques may also be used optimize the detection and measurement of pixels.
Process 200, Identify Segment Color, is diagramed in
Process 300, Build Symbol String, is diagrammed in
Process 400, Build Message String, is diagrammed in
Process 500, Display Message, is diagrammed in
Process 600, Lookup Symbol, is diagrammed in
If the test in process step 650 was true, the symbol string was a match and Process 60 proceeds to process step 660. Where the table symbol is added to the Matching Symbol List. Process 600 then proceeds to process step 665 where the length of the Matching Symbol List is tested. If the length is greater than 1, that is, there has been more than one match for the symbol string identified, Process 600 proceeds to process step 670 where an ambiguity indicator for the symbol string is set. Process 600 then proceeds to process step 675. In process step 665, if the number of matching symbols is 1, process 600 proceeds to process step 675.
Process 600 then proceeds to process step 680 where the table index is compared to the length of the symbol table to determine whether this was the last table entry. If it is not, Process 600 proceeds to process step 620. If it is the last entry, Process 600 proceeds to process step 690 where the symbols from the table that matched the symbol string are returned. Process 600 then ends.
Process 700, Build Symbol Mask, is diagrammed in
Process 800, Message Processing, is an asynchronous process driven by a timer or external interrupt. Process 800 (
The log entry for the message can include a date and timestamp, the message string, and other sensor information that may be both available and relevant (e.g., temperature) for the given device configuration and context. This log entry can be saved in local device memory, external storage device or can be transmitted to a remote site for storage and further processing.
Process 800 then continues to process step 850 where process 500, Display Message is invoked. Process 800 then continues to process step 860 where the current message string is saved as the reference message string for subsequent comparisons.
In process step 870 Process 800 waits for a new message event. This event is device dependent and may be triggered by local sensors or may be externally triggered for purposes of setting the display configuration. Once triggered, Process 800 continues to process step 880 where the message for the event is accessed. This access may be through a lookup table based on possible events, may be supplied by an external source, or may be constructed based upon multiple sources of information available to the device. Process 800 then continues to process step 830.
There are situations where a 1st party controls the messages visible on electronic displays and participates in spatially or temporally separated data transactions or authorization (e.g. access to a venue or right of use) that involve a 2nd party. In many of these situations the 2nd party's participation depends in part on the assumption that the visible message on the display and the data used to effect the transaction are consistent (e.g. they are the same).
For example, retailers (1st parties) typically control the prices visible on electronic shelf labels (“ESLs”) within their stores. Retailers also control the prices actually charged at the point-of-sale during checkout. Consumers (2nd parties) rely upon the information they see on the ESLs to make purchase decisions that are consummated at the point-of-sale with a purchase or payment transaction. Consumers assume that the information visible on the ESLs is consistent with the transaction, but have no way of verifying otherwise once they've left the store. Neither can the retailer.
The prices visible on the ESL at the time of the purchase decision should be the same as those used in purchase transactions, but currently there is no practical way for either consumers or retailers to verify that they were (or were not) in fact. Consumers make mistakes and some perpetrate fraud. And retailers, their employees or IT vendors, or other 3rd parties, also make mistakes and perpetrate fraud.
Consumers “know” (but can be wrong) about what they saw in the store when they made the purchase decision (the prices actually visible on the electronic shelf labels) and later, what they were charged (via printed or electronic receipt—if the detail is provided). And retailers “know” (but can be wrong) about the prices are the same since they control both the information on the electronic shelf label and that used to charge the consumer. Neither party however can be “trusted” by the other. Neither can prove their claim, nor disprove the other party's. Nor can 3rd parties.
A similar dynamic also exists between manufacturers, promotional partners, and retailers related to electronic signage. Manufacturers may for example want in-store consumer facing prices set and maintained at individual locations, across regions or entire chains at precise times synchronous with out-of-store advertising, product launches, time dependent promotions, retailer rebates, inventory price reductions or price protection, revenue sharing, co-op advertising funds etc. And importantly, they only want to pay for the signage/exposure they actually receive. Retailers similarly want to make sure they get paid for the signage/exposure/services they provide.
The same sort of dynamic can exist between advertisers (the parties paying for advertisements or content) and the owners or operators of the media (e.g. owners of digital billboards). It can also exist with payment processors and owners of payment terminals or ATMs. Payment processors have a stake in payment transactions based on transactions that are based on the use of visible information/messages displayed by ATM/payment terminals or visible on remote displays and upon which input into ATM/payment terminals (including mobile devices and smart phones) is dependent.
A similar dynamic can be observed between a patient and the distributor of a drug via a physical container (e.g. medicine bottle, blister pack, vial, fluid bag/container), which includes an electronic visual label. A patient or healthcare professional may check the expiration date of the medicine displayed on the label and decide to ingest/transfuse it or not. Today, if a question was raised as to the validity of such a decision, neither the patient, the drug distributor or a regulatory agency could verify the visible label information at the time of the decision.
Described herein are intelligent visual devices (“IVDs”) that may be non-disposable (e.g. ESLs, electronic bill boards or displays, industrial or medical equipment, smart cards or ‘wearables’) or disposable/consumable (e.g. smart/intelligent labels and packaging, medical “patches”). The IVD may be used to display messaging regarding pricing, for example, as well as product descriptions, dates, warranties, exclusions, alerts, alarms, or other information useful to a consumer or user.
The IVD can be viewed in the context of a closed loop system that can verify that the information presented as a message on an IVD is the same message that was ultimately used in a follow-on data transaction. Herein, the teen “Message Authority” will be an entity that controls the content and message for an IVD. Once the it has been determined what was actually displayed on the IVD, a trusted entity, the Certification Authority, confirms that the intended message is the same as the message as actual visible message used in the data transaction. And depending on the circumstances, if the messages are not the same, the Certification Authority shall determine the actual visible message. In yet another function for the Certification Authority, it may be able to detect tampering, fraud, or system error, and responsive to that, may generate an alarm, an alert, or send messages to any stakeholder.
IVDs are typically uniquely identifiable and addressable, and include a display, for example, bi-stable (e.g. electrophoretic, cholesteric LCD), self-erasing (e.g. LCD, OLED, electrochromic where the visible automatically erases when power is no longer applied to the display) and other known types of displays. The IVD will also have some communication interface that can be wired or wireless, and write-circuitry (hardware, software, firmware or a combination thereof) that ‘writes’ the write-message to the display.
The IVD has determination circuitry that has (1) detection-circuitry that detects the electrical or optical characteristics of the display that correspond to the visible information, for example, the message pixels; (2) conversion circuitry that converts electrical signals corresponding to the electrical or optical characteristics of the message pixels into “states” (e.g. comparative levels of contrast, reflectivity, transmission); and (3) interpretation circuitry that translates or interprets the states of the detected message pixels into higher order symbols or meaning (e.g. into visible alphanumeric characters, symbols, shapes, images etc.). As illustrated in
The IVD also has memory appropriate to the overall purpose and design of the IVD, advantageously persistent/nonvolatile and can preferably be read/write protected. The IVD also has authorization circuitry that manages access to and control over functions of IVD, e.g. to restrict or prevent a Message Authority (described below) from controlling, accessing, or interfering with select functions, data, or communications.
The IVD also has a source of power (that depending on the design or application of the IVD may be continuous or intermittent.) This power source may be internal, such as a battery, capacitor etc., or external such as RF, light, or motion etc. and harvested using an antenna/receiver, photovoltaic or thermoelectric harvester, sensor, mechanical transducer, inductive coupling, a wireless or wired connection etc.
IVDs typically also have timing circuitry (e.g. an internal clock or a reliable external source, particularly in the case of start-up/power reset) and one or more of the following: (1) coding circuitry for performing error detection and correction or encryption/decryption; (2) fault or tamper detection circuitry to manage, for example, a disruption to power; (3) initiating and resetting circuitry; (4) alert or alarm circuitry that can be visible (with optional display element separate from the “display”, e.g. an irreversible electrochromic state indicator), audible (with acoustic emitting device), or haptic (vibration), the alert or alarm may use a wired or wireless transmission; and (5), the IVD may also include geo-positioning or location determining circuitry or information access.
IVDs are configured to receive “write-messages” from a “Message Authority”. A write-message is the message (content) intended to be displayed on an IVD's display and be visible to users (observers). The Message Authority manages (controls) the write-message (content and timing) and its determinants. The Message Authority may transmit appropriately packaged write-messages to one or more IVDs for writing to their displays via embedded write-circuitry. The Message Authority may also set rules typically embedded in an IVD where write-messages are generated responsive to monitored conditions (e.g. temperature, humidity, shock, vibration etc.) or timing events (e.g. elapsed or absolute time), or 3rd party signals etc. The Message Authority may also synchronize or otherwise manage write-messages across multiple IVDs in one or more locations.
The Message Authority also manages, controls and enables, transactions, services and rights associated with write-messages, visible-messages and visible-message-periods (described below). Visible-messages are the messages actually visible on the display of an IVD at a moment in time. Visible-messages are determined by the determination circuitry, which consists of detection circuitry, conversion circuitry and typically interpretation circuitry. A visible-message-period is the period of time a visible-message is continuously visible on the display of an IVD. Visible-message-periods can be determined by comparing visible-messages at different moments in time. The time “window” can be set to reliably determine a continuous (non-changing) visible-message based on the IVD technology.
The Certification Authority spans IVDs and remote systems, and may represent or involve one or more parties. The Certification Authority independently, without influence or interference of the Message Authority (or other unauthorized parties), determines visible-messages and visible-message-periods. Additional exemplary functions of the Certification Authority are Determining (1) Visible-messages before and/or after changes in write-messages or effecting write-circuitry (2) Visible-messages when an event occurs or when an event occurred (3) visible-messages across more than one IVD, or domains of IVDs and (4) diagnosing damaged or faulty displays. The Certification Authority also may algorithmically compare visible-messages with write-messages sent by the Message Authority and may have managing circuitry (functions, authorizations etc.) and information in IVD(s). Further the Certification Authority may provide services (discussed more fully below, where the services are independent of visible-messages and visible-message-periods. The Certification Authority also can be involved in authorizing transactions, services and ‘rights’ dependent on visible-messages and visible-message-periods.
Exemplary services include: (1) Independent determination, verification of visible-messages and visible-message-periods and related events; (2) monitoring, compliance, analysis, analytics, auditing etc.; (3) Fraud detection; (4) notification, alarm/alerts; and (5) risk determination/assessment. Exemplary rights include (1) finance (billing, payment, rebates, credits, settlement, conditional/dynamic pricing etc.); (2) returns, shipments; (3) insurance; (4) Title Issuance, acquisition, transfer; (5) access to information/content and systems, particularly in context of a transaction; (6) participation/membership programs, associations, promotions; (7) Performance based benefits such as successful management of IVDs, visible-messages, trusted services etc.
Visible-Message-Periods
A visible-message-period is the period between known determinations of visible-messages on an IVD. Visible-message-periods are managed by the Certificate Authority (isolated and without interference from the Message Authority). If the IVD's display is bi-stable (visible-messages are stable under normal environmental or operating parameters with or without the continuous application of power) then a visible-message can be considered determinant (known, constant) throughout the visible-message-period. The IVD visible-message can be considered indeterminate if the write-circuitry is active (e.g. there is a new write-message) or hasn't been determined since the last write-message, or the IVD malfunctions or is tampered with.
If the IVD's display is self-erasing or volatile (e.g. visible-messages automatically change if power is disrupted or normal environmental/operating parameters are exceeded) the visible-message is indeterminate during the visible-message-period if power to the display is disrupted or environmental/operating conditions exceeded (e.g. the IVD gets too hot or cold), or the IVD malfunctions, is damaged or tampered. If the IVD is operating correctly, a visible-message only changes in response to the write-circuitry. Therefore, as illustrated in
An explanation for
If the IVD's display is self-erasing (not bistable) and power to the display is disrupted or an environmental/operating parameter exceeded, a visible-message may erase or otherwise change. As illustrated in
An IVD therefore may be advantageously configured with a power back-up (e.g. a battery, charged capacitor etc.) and an automated process to determine the visible-message (using the power back-up) at the time the power is disrupted and store the visible-message along with the time of the event in non-volatile memory (where it is inaccessible and unalterable by other than the Certificate Authority). The last known visible-message may also be automatically transmitted to the Certificate Authority. An IVD may also be advantageously configured so that upon re-starting (power restored) the last know/determined visible-message can be retrieved from memory and automatically written to the display and determined. A sequence of visible-message periods can then be created except for period during the power outage. That would establish a new, known visible-message and associated time from which a variety of determinations could be made; e.g. how long the visible-message was indeterminate (outside a known visible-message period). Alternatively, a new write-message can be automatically generated (e.g. a default, preprogrammed write-message).
An IVD may also be advantageously configured to monitor environmental/operating conditions and in the event of a pending failure (thresholds exceeded) automatically trigger pre and post event-failure determinations to minimize indeterminate visible-message-periods. Note that visible-messages can be determined at any time to detect malfunctions or tampering and ensure continuity and system integrity. The determinations can be made in response to a set schedule (fixed or variable—e.g. time of day) or conditional rules (e.g. changes in monitored conditions), or signals from the Certificate Authority etc.
Extended Certification: Transaction System
Note, here and elsewhere herein, the terms determined, verified and certified may be used somewhat interchangeably depending on the context. A certified visible-message for example, may first be determined (without reference to a write-message) or verified (in reference to a write-message). And both a determined visible-message and a verified visible-message may be certified. An IVD may include certificate circuitry configured to operate without interference or influence of the Message Authority for that purpose. An IVD's certification circuitry may operate (e.g. generate a digital certificate) automatically responsive to the generation of determination messages or other activities within the IVD, events or signals from the Certification Authority. Alternatively, the Certification Authority may generate a digital certificate remote from the IVD (e.g. Certification Authority 922 of
A visible-message may be considered certified if it comes from the Certification Authority (especially if the communication is secure) or by an accompanying digital certificate. Similarly, depending on the context the term visible-message may also refer to digital derivatives, dependents or associated values of the visible-message.
As the term is used herein, a Transaction System is a system/process that executes transactions directly or indirectly affected by visible-messages (or, per the preceding note, their derivatives and dependents). A Transaction System is typically under the direct or indirect control/management of the Message Authority.
An explanation of the numbered data paths for the extended certification system 1010 of
An explanation of the numbered data paths for the extended certification system 1020 of
An explanation of the numbered data paths for the extended certification system 1030 of
An explanation of the numbered data paths for the extended certification system 1040 of
A Transaction System may be distributed and for example, consist of an application on a user's device (e.g. a mobile phone or internet connected terminal) and a remote server to which the user's device is connected. In this example, both the application and the server are directly or indirectly under the control of the Message Authority. Note that in certain circumstances a Transaction System may include a human or human element. Many desired outcomes from the use of IVDs depend on the actions of users and/or their participation in transactions. And in turn, they depend on users trusting the visible-messages on smart labels/packaging, ESLs, video billboards and the like. The perceived risk and willingness of 3rd parties to participate in solutions based directly or indirectly on visible-messages is similarly dependent on them trusting the visible-messages. Note that the perception of independent certification, whether or not it is directly involved in a transaction, increases user participation.
For example, a healthcare professional is more likely to act upon the visible-message on a smart label if they know it is independently certified. The healthcare provider (as well as the emergency service, clinic, or hospital) will likely be held accountable for using (or not using) the visible-message. However at the same time, they will be protected from false accusations arising from a negative outcome due to errors in the message. Regulatory authorities will also likely embrace systems and methods dependent on visible-messages if they are independently certified. All of which helps to drive adoption, participation and positive outcomes.
In another example, customers are more likely to trust a retailer if they know that the visible-messages on ESLs in its stores are certified—and they could unequivocally verify that the prices on their receipts (electronic or printed) are the same as those that were visible on the ESLs in the store when they made their purchase (or within a reasonable period of time from the last write-message). Verification could be done in a variety of ways (e.g. that illustrated
In another example, a dock worker, trucker or anyone proximate with a good with an intelligent label is more likely to trust fair and equitable treatment from being held accountable for properly using visible-messages presented by the label if the visible-messages are certified.
In embodiments of the inventions described herein trust in visible-messages can be achieved by systems and methods that affirmatively communicate to users that the visible-messages are independently certified: that they are as intended (or not) given the circumstances. These systems and methods may be applied to a single IVD (e.g. a smart label or package) or multiple IVDs (e.g. a store of ESLs or public venue with video billboards).
For example, a customer is more likely to trust the visible-message on an ESL if there is an affirmative communication that the visible-message is continuously certified (e.g. a continuous or flashing green light next to the display with the visible message, or a wirelessly transmitted signal to an app on the consumer's mobile phone). In the case of a self-monitoring, self-updating IVD label the absence of an alert/alarm/notification would communicate that the visible-message was certified and trustworthy. The presence of an independent alert/alarm (e.g. a visible, audible, wireless notification) would communicate that the visible-message wasn't certified and thus not trustworthy.
In embodiments of the inventions described herein trust in visible-messages and in turn in the actions or transactions based directly or indirectly on them can be achieved by systems and methods that independently determine or verify (and optionally certify) visible-messages prior to, currently with, or after an action or transaction. Certification with systems and methods such as those described above can be based on the determination of the current visible-message(s) on an IVD(s) associated with a transaction. Alternatively, a determination can be based on a current or previous visible-message-period.
Transactions can be authorized/managed by the Certification Authority by independently (1) providing the Transaction System with certified visible-messages (and their derivatives) upon which the transaction depends (e.g. pricing, promotional offerings) and (2) verifying/certifying the pre-determinants of the transaction received from the Transaction Authority and authorizing their use. For example, the Transaction System may request the current price of an item/SKU from the Certification Authority. The Certification Authority would determine the appropriate IVD (e.g. based on the location in the store where the item/SKU is located), then either communicate with the IVD to determine the then-current visible-message or check the appropriate visible-message-period to determine the current price, and then communicate it to the Transaction System. In embodiments of the inventions described herein, systems and methods authorize actions, transactions, services or rights based on the Certification Authority verifying and certifying visible-messages.
In another example, rights to bill or to receive payment, collect insurance or receive a credit/rebate/discount etc. related to the condition of an item may be authorized subject to independent certification of visible-messages on IVDs such as smart labels or packaging. Similarly, rights to promotional benefits (e.g. digital content, extended warranties or services) may be authorized subject to independently certification of visible-messages on ESLs or video billboards (or even mobile phones—geo targeting).
In another example, transactions, services or rights may be authorized subject to independent certification of visible-messages on multiple IVDs such as an ESL and a POS or payment terminal, or a smart label and a mobile phone or tablet.
In one particularly useful aspect of the intelligent display device, the ability to determine what message is to be written to the display (the write-message) is separated and largely independent from the ability to determine what has actually been displayed as the visible-message on the display. In particular, the intelligent display device uses a Message Authority to determine what message is to be written to the display (the write-message), and to take the steps necessary to cause that message (the visible-message) to be displayed to a consumer or other user. A Certification Authority is used to manage the determination of what was actually visibly displayed. In this way, the Certification Authority is able to make a determination of what has actually been displayed (the determined-message, without influence or tampering from anyone associated with the Message Authority.
By separating the control over (1) what is intended to be displayed from the (2) determination of what was actually displayed, a great deal of trust can be created over the visible-message, as well as any transaction relying upon the accurate display of that message. As described in detail above, the intelligent display device is constructed to receive instructions regarding writing information from the Message Authority and converting that information into a write-message that is to be displayed on the message display. In some cases, the Message Authority may be able to write the write-message immediately to the display, and in other cases the Message Authority may need the authorization of others, for example via the rules set by the Certification Authority, prior to start the write process. Also, the write-message that is to be displayed may be received from an external source, or it may be generated internal to the intelligent display device.
In enabling trusted commercial or non-commercial transactions, events and actions on the part of customers and other users, the Certification Authority is of central importance. It is the Certification Authority that acts as a trusted middleman to assure and confirm that the message that should have been displayed by the Message Authority (the write-message) is the actual message that was visible to, for example, a consumer or other user (the visible-message). In order to extend the flexibility and usefulness of the Certification Authority, the Certification Authority may operate according to a set of predefined rules set in intelligent display device. The sophistication and complexity of these rules may be adjusted according to application specific requirements.
In one example, a Message Authority receives a write-message that is to be displayed to a message display on an intelligent display device. In some cases, this write-message may have been received from an external source, and in some cases, it may have been generated internally within the intelligent display device. Such internal messages may include messages regarding or associated with time, date, environmental conditions, or other aspects of the intelligent display device itself. In a simple case, the rules for the Certification Authority causes the intelligent display device on behalf of the Message Authority to immediately write the write-message to the message display. Upon completion of writing the write-message, the Certification Authority is notified that a write to the display has been completed. Immediately thereafter, the Certification Authority causes the determination circuitry to determine what message is presently visible on the display. Importantly, the intelligent display device is configured such that the determination circuitry is not influenced or under any control of the Message Authority.
The Message Authority communicates the write-message to the Certification Authority, and as described above, the Certification Authority determines the current visible-message and thereby, generates the determined message. Accordingly, it is a very straightforward process for the Certification Authority to confirm whether or not the visible-message was the same message as the one provided by the Message Authority. That determination, of whether the messages are the same, can then be communicated to third parties, used by the Certification Authority for further processes, or used internally to the intelligent display device.
In some cases, it may be desirable that the Certification Authority's rules provide more oversight to the message display process. In such an example, when the Message Authority desires to write a new message, it first must authorized by the Certification Authority prior to doing a write function. Although there are several examples of where this would be useful, a particularly valuable example is where the message display already has a visible-message being presented. The intelligent display device receives from the Message Authority an instruction to write a new message to the display. In this case, the Certification Authority has set rules for the operation of the write-circuitry, where the write-circuitry must be an authorized by the rules of the Certification Authority prior to changing the current visible-message to the new message. In one example, when the Certification Authority is notified that that the display is about to be updated, the Certification Authority rules may cause a determination to be made of what is actually currently being displayed. Once the current actual display information is captured and the visible information is determined, preferably along with time information, then the Certification Authority will enable the write of the new message to the display to proceed. As described above, when the Certification Authority is made aware by that the new message has been written to the message display, the Certification Authority will then make another determination to assure that the new message has been properly and visibly displayed.
In contrast to most known display devices, the intelligent display device is not focused on fast refresh and fast message display. Instead, it is focused on confidently determining what the user actually was able to see, verifying that the actual visible-message is, or was, as intended, and certifying the display information can be trusted by all parties involved. Accordingly, when the Certification Authority has verified that the display had the correct message, or did not have the correct message, or otherwise determined what the visible message actually is (or was) it may generate a digital certificate. This digital certificate may be used internally for making adjustments to the intelligent display device, or may be communicated externally so others may make appropriate commercial decisions. In some cases, it is desirable to encrypt the digital certificate with information specific to the intelligent display device, such as its serial number, geo-location, date, time, or other information that can identify the specific message display.
In some cases, there may be a disruption to the message display, such as a loss of power, an error condition, a forced reset, a shock, or a detection of an auto reboot. In such a case, the Certification Authority may perform a determination immediately after the disruption. Again, this can be used to enhance trust, reinforced with the generation of a digital certificate. In another example, the Message Authority or another entity, may desire that a determination be made at a specific time or upon the occurrence of a specific event. Accordingly, the Certification Authority may cause a determination to be made responsive to the receipt of an internal or external signal.
In another example, particularly useful when a display is intended to hold a message for an extended period of time, the Certification Authority makes a first determination to what is actually visibly displayed when a write-message is first displayed, and then make a second determination at a second, later, time to confirm that the message is still properly and visibly displayed. In some cases, this later time may be a time just prior to when the next write-message is ready to be displayed and replace the current visible-message. In particular, the Certification Authority will compare the determined first visible-message to the determined second visible-message, to verify that the visible-message is the same at the start and the end of the message time period (the visible-message-period). The Certification Authority may also detect or determine if some disruption has occurred during the visible-message- period. If so, the visible-message may be indeterminate, and its uncertain state should be reflected in a digital certificate generated by the Certificate Authority. Further, the Certificate Authority may cause an alarm or alert to be set, or cause a message to be displayed on the intelligent display device, or a message to be sent to a remote location. Accordingly, when there is uncertainty regarding the visible information, all parties are typically notified. In such a manner, the intelligent label device generates high quality visible information and related message data that can be used to enable trusted commercial transactions.
The structures and functions for optically determining a message on a display have been fully set forth in co-pending U.S. patent application Ser. No. 15/368,622, entitled “Optically Determining Messages on a Display,” filed Dec. 4, 2016, and which is incorporated herein as if set forth in its entirety. For completeness, a summary description is provided below regarding the structures for making an optical determination. Of particular relevance is the photodetection layer, which as described herein, may also in whole or in part be used as a photovoltaic power source for the intelligent display device.
A display device, as defined in co-pending U.S. patent application Ser. No. 15/368,622, comprises a display layer and a light detection layer. Devices may also have a light source layer. These functional ‘layers’ may be configured in different ways and in different combinations depending in part on their respective reflective, transreflective or transmissive properties. They may also share common elements (e.g. common electrodes). The term ‘layer’ should be construed broadly to encompass configurations other than those where the functions ascribed to the terms above are literally layered. Of particular interest are configurations where the display layer, light detection layer and light source layer, as well as, the assembled device, are flexible. Devices however, and their components, may also be semi-rigid and rigid. Devices may also include electronics, methods and systems described herein.
The display layer displays the message and may be any of different types including, but not limited to, electrophoretic, liquid crystal, plasma, OLED, and electrochromic. Of particular interest are displays (display layers) that are bi-stable or irreversible. Display layers may be further distinguished in accordance with their ability to reflect/absorb or pass/block light. An example of the latter that is of particular interest are electrophoretic displays comprising transparent electrodes where the charged particles may be positioned so that in one state they block light from passing, and in a second state they are moved out of the light path, and allow light to pass.
A light detection layer is typically sized appropriately to detect/measure light associated with the state of the display pixels and optionally, other areas such as that for detecting/measuring ambient light. A light detection layer (photoactive sensor) can be made of photovoltaic materials, light harvesting proteins, or other photoactive compounds. Preferred photovoltaic materials include organic photovoltaic materials (OPV) for ease of roll-to-roll manufacturing and optical properties (e.g. high transparency).
An exemplary embodiment of a light detection layer consists of a transparent electrode layer of ITO, an organic photovoltaic material based on for example Poly 3-hexylthiophene (P3HT) and an electrode layer (transparent or non-transparent) such as ITO, PEDOT:PSS, graphene, a metal conductor (e.g. Al), or a combination thereof. Of interest are organic photovoltaic devices that are near transparent or semitransparent (see e.g. US Pub. No. US20140084266 “Semi-transparent, transparent, stacked and top-illuminated organic photovoltaic devices,” and US20120186623 “Transparent Photovoltaic Cells,” and U.S. Pat. No. 5,176,758 “Translucent Photovoltaic Sheet Materials and Panels”). Bacteriorhodopsin (see, e.g., “Photoelectric response of polarization sensitive bacteriorhodopsin films,” Q. Li et al., Biosensors and Bioelectronics 19 (2004) 869-874, and included references) is a preferred light harvesting protein for the photoactive layer. In certain devices a light detection layer (e.g. photovoltaic photoactive sensor) also may serve a dual purpose and be used for message determination/verification and for energy harvesting.
In an exemplary display device 1110, shown in
The photoactive layer in the above configurations can be made of photovoltaic materials, light harvesting proteins, or other photoactive compounds. Preferred photovoltaic materials include organic photovoltaic materials (OPV) for ease of roll-to-roll manufacturing and with optical properties of high transparency to minimize the impact of the display readability. Of particular interest are organic photovoltaic devices that are near transparent or semitransparent developed primarily for automotive and building window applications (see e.g. US Pub. No. US20140084266 “Semi-transparent, transparent, stacked and top-illuminated organic photovoltaic devices,” and US20120186623 “Transparent Photovoltaic Cells,” and U.S. Pat. No. 5,176,758 “Translucent Photovoltaic Sheet Materials and Panels”). Bacteriorhodopsin (see, e.g., “Photoelectric response of polarization sensitive bacteriorhodopsin films,” Q. Li et al., Biosensors and Bioelectronics 19 (2004) 869-874, and included references) is a preferred light harvesting protein for the photoactive layer.
While particular preferred and alternative embodiments of the present intention have been disclosed, it will be appreciated that many various modifications and extensions of the above described technology may be implemented using the teaching of this invention. All such modifications and extensions are intended to be included within the true spirit and scope of the appended claims.
This application is a continuation in part to U.S. patent application Ser. No. 14/927,098, entitled “Symbol Verification for an Intelligent Label Device,” filed Oct. 29, 2015, which claims priority to U.S. provisional patent application No. 62/199,653, filed Jul. 31, 2015 and entitled “Verification of Messages Displayed with Electro-Optic Devices,” both of which are incorporated herein in their entirety. This application also claims priority to U.S. patent application Ser. No. 15/368,622, entitled “Optically Determining Messages on a Display,” filed Dec. 4, 2016, and which is incorporated herein as if set forth in its entirety. This application also claims priority to U.S. provisional application No. 62/341,768, entitled “Systems and Methods for Independently Determining Visible Messages on Intelligent Visual Devices,” filed May 26, 2016, which is incorporated herin in its entirety. This application is related to U.S. patent application Ser. No. 14/479,055, filed Sep. 5, 2014, and entitled “An Intelligent Label device and Method,” which is incorporated herein in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 62199653 | Jul 2015 | US | |
| 62341768 | May 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14927098 | Oct 2015 | US |
| Child | 15407463 | US | |
| Parent | 15368622 | Dec 2016 | US |
| Child | 14927098 | US |
