The invention is related to information handling, and more particularly, to ways of using handheld devices for transporting information from one work area to another, such as in an office environment.
It has almost become a truism that in the Information Age, office workers have become enveloped in information, with some of this information being central to immediate and ongoing tasks, and other information being of less immediate importance. In most offices, information is scattered about in many places. Desk surfaces may be piled high with papers and books; in addition, computer screens, each of which may display different sorts of information, are ubiquitous. Even a single computer screen may be crowded with information, in which e-mail, word processing, and programming applications all occupy a portion of the screen, or alternatively, a different “window”. Furthermore, the perimeter of the typical computer screen, as well as the office workspace generally, are often occupied by numerous “sticky” notes to remind the user of various events, things to do, phone numbers, and so on. Indeed, notes on paper do have the advantage of being easy to use when it comes to certain kinds of tasks, such as reading, cross-referencing and annotating information, physically moving information, and posting information (to get someone's attention or remind someone of something). For example, posting information may involve placing information in a place that is physically significant to the user, such as next to the phone.
As the typical work environment has continued to rely on information scattered among available surfaces and objects, managing this information has proven to be cumbersome. Unfortunately, capturing information that is physically strewn about an office does not easily lend itself to an electronic solution. In particular, the notes posted by office workers along the perimeters of their computer screens have so far eluded a means of being handled electronically. There remains a need for handling such information electronically, as well as a need to free electronic information from the confines of an isolated computer screen, thereby allowing information to be distributed more naturally and easily throughout the office environment.
Preferred embodiments of the present invention address these needs through the use of handheld display screens (such as liquid crystal displays—LCDs) that allow information to be attached to a variety of surfaces, e.g., on a wall or around the edge of a monitor. These display screens have specialized interactivity and functionality and act as dedicated information tools. By placing these handheld display devices along the perimeter of a computer screen, they lend themselves to receiving images displayed on the computer screen and thereby free information that is peripheral to a user's immediate concerns from an already crowded screen. In this way, the handheld display screens can be used to effectively move information off a computer screen and dynamically into the physical world.
Such display screens are well-suited to function as reminders or otherwise provide awareness of information. Thus, preferred embodiments of these display screens may be used to satisfy tasks normally performed with paper, such as posting notes around the office, putting information in one's pocket for later reference, or giving away (visually or informationally enhanced) business cards. Preferred embodiments of these display screens require only the most basic interaction capabilities, such as copying data to and from the display screen, paging through information displayed on the display screen, and pausing the display update, but they need not have computational power or a complex user interface.
One embodiment of the invention is a monitor that has a screen having a perimeter that defines the extent of the screen, and at least one electronic device that generates electronic signals for producing images on the screen. The monitor also includes a housing connected to the perimeter of the screen, in which the housing includes a data port for receiving an electronic display unit. The data port is located near the screen perimeter and is configured such that when an electronic display unit is connected to the data port, both the electronic display unit and the screen are visible to a user positioned in front of the screen. The data port is preferably configured to receive an electronic display unit that is, when connected to the data port, substantially parallel to the screen. In one preferred embodiment, the data port is located within 5 cm of the screen perimeter. The housing may include 3 or more such data ports.
Another embodiment of the invention is a monitor that includes a screen having a perimeter that defines the extent of the screen, and at least one electronic device that generates electronic signals for producing images on the screen. The monitor also includes a housing connected to the screen perimeter, with this housing including at least 2 data ports for receiving an electronic display unit. Data ports are located near the screen perimeter. The housing may include at least 4 data ports located near the screen perimeter, e.g., within 2 cm of the screen perimeter.
One implementation of the invention is a method of copying images displayed on a monitor to a handheld electronic display unit, in which the display unit and the monitor include respective screens for displaying images. The method includes electronically detecting the presence of the display unit in response to it being brought to the monitor by a user. The method further includes providing an indication to the user that the display unit has been detected, and electronically copying a user-selected image that is displayed on the monitor's screen to the display unit, wherein the selected image is displayed on the display unit's screen upon completion of the copying. This indication may include visual feedback on the screen. The image as it is displayed on the display unit's screen is preferably large enough that a user can view and understand its contents simply by looking at the display unit's screen, and the selected image is advantageously displayed both on the monitor's screen and the display unit's screen upon completion of the copying. In one preferred embodiment, the monitor includes a port for receiving the display unit, in which the display unit is detected in response to being connected to the port. The port may be located near an edge of the screen. In one preferred implementation, the presence of the display unit is electronically detected in response to being brought within 5 cm of the monitor's screen, and in another implementation, the display unit is electronically detected in response to being brought within 2 cm of the monitor's screen. In one preferred implementation, this detection may occur wirelessly, and data and images may be transferred wirelessly from (or to) a processor connected to the monitor.
The method may further include, after a user has brought the display unit to a second monitor having a screen, electronically copying the selected image from the display unit onto the screen of the second monitor. This may involve the display unit being connected to a port on the second monitor. The method may include providing an indication on the second monitor's screen that the display unit has been connected to the port on the second monitor, and providing a graphical user interface on the second monitor's screen to permit the user to copy the selected image to the second monitor's screen.
In other implementations, there are provided handheld electronic display units to be used in carrying out any of the methods herein.
In still other implementations, there are provided computer program products for carrying out any of the methods herein.
Overview
Preferred implementations of the invention provide a user with the ability to quickly and seamlessly move information between a hand held device and computers/monitors. By way of example, and with respect to
As discussed in more detail below, the display unit 56 has a connector that mates with the data port 50 in the monitor's housing 44. The data port 50 and the display unit 56 preferably mate in such a way that the display unit is clearly visible to a user positioned in front of the screen 38. For example, when the display unit 56 is connected to the data port 50, the face of the display unit is preferably aligned parallel with the face of the screen 38. In this way, the user can easily view what is displayed on the screen 38 and the display unit 56 at the same time. As shown in
The apparatus shown in
The display unit 56 may then be transported by the user to another location, e.g., to a place where it is conspicuous (such as on a refrigerator door) to serve as a reminder to the user; in this application, the display unit functions essentially as a “note”. (To this end, the back side of the display unit 56 may include appropriate attachment means, such as adhesive, Velcro™, or a magnet.) Alternatively, the display unit 56 may be brought to the screen of a second monitor (not shown) and connected to a data port in that monitor's housing (similar to the configuration shown in
Hardware Architecture
In addition to the software architecture described below, various hardware is used for the preferred embodiments and implementations of the invention described herein. The hardware involved includes the display unit 56, the monitor 20, the processor 62, and any associated peripheral devices in electronic communication with the processor, such as a mouse 154.
The display unit 56 itself includes a screen 60, a battery 156 (that is preferably wired in parallel with the screen 60), as well as several electronic components in electrical communication with each other that are designated collectively in
The battery 156 is a 1400 mA lithium polymer battery. Recharging of the battery 156 is accomplished by supplying DC 5 volts at 2.0 A via the female and male power connectors 104 and 112. The power wires 130, 134 are connected to a transformer (not shown) that in turn is supplied by a 100–240 volt AC power source (e.g., via a standard outlet). This ensures that the batteries of the display units 56 are being recharged whenever the display units are connected to a port 50. The user connects the display unit 56 to one of the ports 50 on the monitor 20 (see
The display unit 56 shown in
As shown in
Copying Images and Data to the Display Unit 56
Preferred implementations are now described in connection with copying images appearing on the screen 38 of a monitor 20 (and residing electronically in the processor 62, which is in communication with the monitor) to the display unit 56. The electronic display unit 56 includes memory; if this memory is “blank” prior to use, no image appears on the screen 60. When the display unit 56 is initially connected to the port 50, the screen 60 of the display unit preferably remains blank. Following this connection, the display unit 56 and the host software in the processor 62 exchange configuration information. The display unit 56 reports its display size, the image format required for display, memory size, and memory/display contents (if any) to the processor's software. The processor's software reports the orientation of the display unit 56 with respect to the monitor 20, which is determined by the location of that port to which the display unit is connected. For example, if the display unit 56 is attached to the bottom of the monitor 20, then the orientation of the image on the display unit 56 is ideally rotated 180 degrees with respect to what it would be if the display unit were attached to the top of the monitor.
The processor 62 provides feedback to the user when the display unit 56 is connected to the data port 50. This feedback is preferably in the form of the graphical user interface 70 or graphical tab (see
A user may now send an image (appearing on a web page, for example) or, alternatively, data (such as a file in a directory or a section of text in an open application) to the display unit 56 after selecting the image (or data) using one of several selection methods. First, the user may use a standard method offered by the operating system to select the image. Using a cursor 159 on the monitor's screen 38 controlled by the mouse 154, the user can then drag the selected information to the graphical tab 70 and drop the information anywhere onto the tab (i.e., a “drag-and-drop” procedure). In a preferred implementation, the graphical tab 70 appears on the screen 38 at a location adjacent or nearly adjacent the display unit 56, thereby allowing the user to conveniently move information from anywhere on the monitor screen 38 to the peripheral display unit 56. Once the user positions the cursor 159 over the graphical tab 70, the tab may advantageously increase its brightness, indicating to the user that the image may now be copied to the display unit 56 (e.g., by releasing the mouse 154).
In another selection method, the user may cut a selected image into the clipboard, move the cursor 159 to the graphical tab 70, and paste the selected image from the clipboard into the graphical tab (i.e., a cut-and-paste procedure).
In yet another selection method, the user may select an image by selecting a window in a windows style GUI, and move the selected window until it overlaps the graphical tab 70, thereby transferring the contents of the selected window to the display unit 56 and its screen 60 (i.e., a snapshot window procedure). This selection method (as well as the others described herein) can be repeated: By changing the contents of the selection window, and then overlapping the selection window with the graphical tab 70, the newly selected contents are then sent to the display unit 56 and its screen 60, replacing the previous contents.
Additional selection methods and options can be made available to the user through the “Select” label 168 on the top row of menu options. For example, an option can be offered to the user in which the contents of a given window are always simultaneously displayed on the screen 60 of the display unit 56 and the monitor's screen 38, even when this given window is obscured by other windows on the screen 38 of the monitor 20. By selecting this option from a pull-down menu (not shown) under the select label 168, the cursor turns into a crosshair (not shown) which can then be used to define a window that about a desired image. By left clicking (with the mouse 154) on a window so selected, the desired image is transferred to the display unit 60, at which point the crosshair returns to a cursor. Such a selection option allows the user to keep information in a window always visible to him or her, since the image so selected is viewable to the user on the screen 60 of the display unit 56. Still other selection methods may occur to those skilled in the art.
Any of the aforementioned methods can be used to transfer an image (or, more generally, data) displayed on the monitor's screen 38 from the processor 62 to the display unit 56. Once an image is selected, it is preferably converted into at least two formats: an image format for display on the screen 60 of the display unit 56, as well as a native format to be held in the memory of the display unit. For example, if text is selected by the user, the text is advantageously converted to a Bitmap image file, and this Bitmap image file and the associated ASCII text file(s) are transferred over to the display unit 56 via the serial connection (defined by the female and male serial port connectors 100 and 108). The Bitmap or other image file is preferably sent first so that the display unit 56 can display it as soon as possible. The display unit 56 detects that new data has been sent to it and sends the image file directly to the screen 60, with any associated files (e.g., ASCII text files) then being put into the memory of the display unit 56. Any image that had been displayed on the screen 60 of the display unit 56 or had been held in memory is replaced with the new information. It may also be desirable to convert data into several different formats. For example, a Microsoft Word document (i.e., .doc file) can be converted into Rich Text Format, ASCII format, and Bitmap format. All of these formats can then be transferred to the display unit 56 for later use by the user.
Thus, when the user selects an image or other information to be sent to the display unit 56, the information is displayed on the screen 60 almost immediately, with the image being visible on both the monitor's screen 38 and the display unit's screen 56. The graphical tab 70 may also advantageously show the formats of the information currently held in the display unit 56, as indicated by the textual labels “Bitmap”, “Plain Text”, “JPEG”, and “GIF” shown in
Copying Images and Data from the Display Unit 56 to a Processor/Monitor
An image (or other information) appearing on the screen 60 of the display unit 56 may be copied from the display unit to a processor, so that the image can then be displayed on the screen of a monitor in electronic communication with that processor. This may be done by connecting the display unit 56 to a data port 50 on a monitor, e.g., a (second) monitor other than the one to which the display unit was connected when the image was copied to the display unit. Alternatively, the display unit 56 can be connected to any compatible serial port, such as one mounted in the CPU housing of a computer. After the display unit 56 has been connected to such a port, the display unit and the software in the processor exchange configuration information, as outlined above in connection with copying data to the display unit. A graphical tab like the tab 70 then preferably appears on the screen of the second monitor, providing an indication to the user that the display unit 56 has been connected. If the display unit has data in its memory, the graphical tab advantageously shows the data formats available on the display unit 56 using text labels (such as those shown in
An image can be copied from the display unit 56 using cut-and-paste, drag-and drop, and pull-down menu interactions, analogous to those described above in connection with copying an image to the display unit 56. Preferred methods for performing these copy operations are now described with reference to the graphical tab 70 shown in
The user can also work with information in the display unit 56 as a file using a File pull-down menu 176, as shown in
Software Architecture
The architecture used for the preferred embodiments and implementations of the invention herein is now described, including that for the “client” software of the display unit 56 and the “host” software of the processor 62/monitor 20. The client software used was built on a client-host model similar to that used in many network applications, in which client software runs on a client computer that is connected over a network to a server (i.e., the host). In the embodiments and implementations of the invention described herein, the client software runs on the display unit 56, and the host software runs on the processor 62 that drives the monitor 20; thus, the display unit 56 functions as the client and the processor 62 as the host. The software on the display unit 56 advantageously performs several functions, including: 1) displaying an image of the data transferred to it; 2) detecting when new data is sent from the processor 62; 3) detecting requests for data from the processor; and 4) storing and sending configuration data. On the other hand, the software on the processor 62 advantageously performs several functions, including: 1) detecting the presence of the display unit 56 in the port 50; 2) establishing a connection with the display unit; 3) requesting data from the display unit; 4) sending data to the display unit; 5) providing a user interface to the display unit; 6) providing feedback about the state of the display unit system; and 7) translating data prior to sending it to the display unit.
The software on the processor 62 was written in Sun's Java version 1.3 Standard Edition for the Microsoft Windows (98/NT/2000) operating system. The software on the display unit 56 was Sun's Personal Java for the Microsoft Windows CE operating system. This software design allows for relatively easy portability to new devices and operating systems. It also allows the host and client devices to operate independently so that, for instance, if the (host) processor 62 loses power, any display units 56 connected to the monitor 20 will continue to operate and display information.
1. Client Software (
As indicated in steps 310 and 315, the (client) software on the display unit 56 continuously listens for data sent through the serial port, via the (female and male) serial port connectors 100 and 108. A determination is made whether the data received are in the form of a control signal byte or a data signal byte (step 320), with the client software handling each possibility accordingly. Receiving a control signal indicates that the processor 62 (host) is sending a request, whereas receiving a data signal indicates that data are being sent.
As shown by step 330, if a control signal is being sent, a determination is made whether i) the request is for data from the display unit's memory (this possibility is indicated in step 330 of
On the other hand, if the data received are in the form of a data signal byte (“Data”, step 320), this is an indication that the processor 62 is sending data to be stored in the display unit's memory. If the data being sent are determined (step 360) to be images data of the correct format (e.g., Bitmap), then those data are used to refresh the screen 60 of the display unit 56 (step 362). The refresh process is performed by obtaining the image data, obtaining the current orientation from memory, and mapping the image data in the correct orientation to a temporary memory space (e.g., an application memory in which the image is assembled; this process is known as double-buffering). Finally, the image in the temporary memory space is swapped into DMA for a clean transition to the new image. After being displayed, the image is sent to memory to be stored (step 370). All other incoming data (step 370, “No”), such as non-image data, are sent directly to memory.
2. Host Software (
As indicated in
Next, configuration information from application memory is retrieved, the graphical tab 70 is processed and displayed (step 450) on the monitor 20 at the location of the newly occupied port 50. Using Java, the graphical tab 70 is a window object similar to any other window in a GUI, but without a title bar or border. The graphical tab 70 is preferably set to always be “on-top”, so that it cannot be obscured by other windows. Java facilitates including the same user interface functions to the graphical tab 70 that all other windows have, including drag-and-drop (on and off the tab), cut-and-paste, and select. Adding these standard user interface capabilities is done by creating an event listener for the graphical tab 70. For each display unit 56 in communication with the processor 62 via a port 50, a new graphical tab window object is created and shown on the monitor 20, and assigned an event listener. The event listener waits for some user action to occur (step 460) and then signals the host application, if appropriate (“Yes” of step 470). How these events are processed (step 480) is covered in greater detail below (see
After the graphical tab 70 is deployed, the host application polls the display unit 56 for data, i.e., whether configuration data (information) or file data are being transmitted by the display unit 56 in response to the poll (step 500). If either kind of data is found, then the data are stored in application memory (step 510). If no data are found, then the host application sends an acknowledgment (ACK) request to the display unit 56 (step 520). The host application waits for an acknowledgment response, and if a response is received (“Yes” of step 530), then the host application continues polling. If no response is received (“No” of step 530), the host application assumes the display unit 56 has been disconnected. The port-table is updated to reflect the fact that the previously occupied port is now unoccupied, and the associated graphical tab 70 is removed from the monitor 20 (step 540). Throughout this process, the host application runs three different threads, as indicated by steps 410, 490, and 460, namely, (a) an empty port polling thread to detect if a display unit 56 has been attached to the (host) processor 62; (b) an occupied port polling thread to collect data and detect if a display unit has been disconnected; and (c) an event listener thread to detect user interface actions. Information about which ports 50 are empty and which are occupied is kept in the port-table in application memory.
User interactions with the graphical tab 70 are now described with respect to
As indicated in step 610 (“send”), the user may send information or data to the display unit 56 by one of number of procedures discussed previously, including dragging a filename or data (e.g., with a cursor) to the graphical tab 70, cutting data into the clipboard and pasting it into the graphical tab, and opening a file with the File pull-down menu. Regardless of which procedure is used to send data to the display unit 56, the host application processes the request in the same way. First, the selected file or data are captured (step 680). A determination is made whether the data are already image data in the correct Bitmap format (step 690), and if not, these data are converted into that format (step 700). This conversion (step 700) is easier if the data are already in an image format; if, however, the data are plain text or rich text, the text is preferably fit into an area that is the same size as the screen 60 of the display unit 56. This is done by performing word-wrap and adjusting the font size until a maximum amount of text is visible on the screen 60 of the display unit 56. In the case of rich text data, the layout is preserved (e.g., relative font sizes, paragraphs, underlines). If the image data generated from the text data are still larger than the display area of the screen 60, then the image is left at that size and only the upper right corner of the image is seen on the display unit 56. If the image data were not converted form text, but rather another image, then the image data are transformed according to the configuration parameters stored in the port-table (step 710). The transformation step may be unnecessary if the data are copied directly to the display unit 56 (e.g., by using a selection square 164 as discussed previously). If the image data are larger than the screen 60 of the display unit 56, then the transformer tries to shrink the image by up to 20% in the x and y dimensions. The transformer will also try to warp the image space so that “blank” areas are reduced in size. (Blank areas are locations in the image with small changes in pixel intensities.) If the image is still larger than the screen 60, the image is left at that size, and only the upper-right corner is seen on the display. After conversion (if this is necessary, see step 700) and transformation (step 710), the image data are sent to the display unit 56 (step 720). After the display unit 56 receives all the data, the data are displayed (preferably) immediately. Next, the host application converts the selected data into any other desired formats (step 730) and sends those data to the display unit 56 (step 740) for storage in the display unit's memory—the image data and all formats are saved in the local application memory (step 750) on the host (processor 62).
As indicated in step 610 (“select”), the user may also request a special selection method from the graphical tab 70. If the selection square 164 is chosen (which is represented later in the flowchart as decision 785), the cursor 159 is converted to the image of a selection band that is the size of the display unit 56 in pixels and has the same orientation (step 770). The user may move this selection square (164) around the screen 38 of the monitor 20 just like the cursor 159. A system-hook is activated that allows the host application to intercept system events such as mouse clicks. If the user right-clicks (step 780, “right-click”) when the selection square 164 is displayed, the cursor 159 reappears and nothing is selected. If the user left-clicks (step 780, “left click”), then the pixels in the selection square 164 are copied and this copied image data is sent to the display unit 56, corresponding to the “square” option shown in the selection decision 785 (a decision which has been previously made by the user). After the image has been copied, the selection square 164 is replaced by the cursor (step 800).
If the user wishes to select the contents of a window using the graphical tab 70 (corresponding to the “window” option shown in the selection decision 785), the cursor is converted to appear as a targeting crosshair (step 770). If the user right-clicks, the crosshair returns to the normal cursor appearance (step 800), and nothing is selected. If the user left-clicks, the host application obtains the identification (ID) number of the window over which the cursor was positioned. A window listener is then launched (step 810) that watches the window's repaint events. When a repaint event occurs after 2000 ms, the pixels in the window are copied (step 790) and sent as image data to the display unit 56 after conversion and transformation.
Other Embodiments
As an alternative to the fixed data ports 50 along the perimeter of the screen 38 of the monitor 20, wireless data links may be provided. The wireless ports can be in principle unlimited in number and can assume any location along the perimeter of the screen 38. In this case, display units (not shown) having wireless capability are (wirelessly) connected to the processor 62. These ID codes correspond to physical locations along the perimeter of the screen 38. The host system has ID codes that are transmitted through blinking patterns of the pixels on the screen 38, with a sensor in the wireless display unit receiving commands and information from the processor 62. This sensor, or optical input device, is integrated with the wireless display unit and can obtain the ID code corresponding to that location on the screen 38. In this implementation, the wireless display units can be hung directly in front of the screen 38 (e.g., within 2 cm of the screen's surface) by attaching them to the monitor 20 (e.g., magnetically) so that they extend far enough beyond the perimeter of the screen 38 to obtain an ID code through the blinking pixels. When this ID code is then wirelessly transmitted from the wireless display unit to the host system (e.g., the processor 62), the host system responds by providing graphical feedback and a user interaction object on the screen 38 at the location of the ID code. The data transfers also occur through the wireless link. Using this wireless method, the user can physically hold a display unit configured for wireless communication in front of the monitor 20, drag an image or other data from the monitor's GUI to the graphical feedback object (such as a graphical tab 70), thereby copying over data to the display unit 56. Similarly, images and other data may be copied from such a wireless display unit to another processor and its monitor.
The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is therefore indicated by the appended claims rather than the foregoing description. All changes within the meaning and range of equivalency of the claims are to be embraced within that scope.
In preferred embodiments of the invention, there is provided media encoded with executable program code to effect any of the methods described herein. This code contains executable instructions that may reside, for example, in the random access memory (RAM) of a processor, or on a hard drive or optical drive of a processor. The instructions may be stored on a magnetic or optical disk or diskette, a disk drive, magnetic tape, electronic read-only memory, or other appropriate data storage device. In preferred embodiments, this program code may be read by a digital processing apparatus such as a processor or computer for performing any one or more of the methods disclosed herein.
Number | Name | Date | Kind |
---|---|---|---|
5530235 | Stefik et al. | Jun 1996 | A |
5613095 | Moss et al. | Mar 1997 | A |
5769374 | Martin et al. | Jun 1998 | A |
5805415 | Tran et al. | Sep 1998 | A |
5821931 | Berquist et al. | Oct 1998 | A |
5883621 | Iwamura | Mar 1999 | A |
5920706 | Ono | Jul 1999 | A |
5948091 | Kerigan et al. | Sep 1999 | A |
5978591 | Bartholomew et al. | Nov 1999 | A |
6037954 | McMahon | Mar 2000 | A |
6060993 | Cohen | May 2000 | A |
6151021 | Berquist et al. | Nov 2000 | A |
6236330 | Cohen | May 2001 | B1 |
6313877 | Anderson | Nov 2001 | B1 |
6522309 | Weber | Feb 2003 | B1 |
6545669 | Kinawi et al. | Apr 2003 | B1 |
6573913 | Butler et al. | Jun 2003 | B1 |
6766382 | Madden et al. | Jul 2004 | B1 |
6784855 | Matthews et al. | Aug 2004 | B1 |
20020049784 | Bauchot | Apr 2002 | A1 |
20020140688 | Steinberg et al. | Oct 2002 | A1 |
20020154120 | Cullimore et al. | Oct 2002 | A1 |
Number | Date | Country | |
---|---|---|---|
20030227438 A1 | Dec 2003 | US |