This application is based upon and claims the benefit of priority from the prior Japanese Patent Application No. 2006-91826 filed on Mar. 29, 2006, the entire contents of which are incorporated herein by reference.
1. Field of the Invention
The present invention relates to a screen transmission device, screen display method and computer readable medium, and more particularly, to a window display position control technology in a multi-display system which displays a computer screen of a PC (Personal Computer) or the like by coordinating a plurality of displays.
2. Related Art
Conventionally, as a system for displaying one computer screen of a PC or the like using a plurality of displays, there is a multi-monitor function supported as standard by Windows (registered trademark). When a display card for multi-monitor is used or a plurality of display cards are pushed into the PC simultaneously and the OS (Operating System) recognizes the plurality of displays, screens are displayed as one big desktop on the plurality of displays. The user can use a wider work space when using a plurality of displays than the case where the user is working only with one display. For example, when the user wants to see a plurality of documents simultaneously, document windows do not overlap with one another or windows are arranged so as not to overlap with one another, and therefore the windows can be displayed simultaneously without being reduced in size.
A system whereby a plurality of displays cooperate with one another for displays is described in JP-A 2003-61140(KOKAI). According to JP-A 2003-61140(KOKAI), when a display instruction is received from a user, the system judges whether or not all the instructed display contents can be displayed on a full screen at a time. When all the instructed display contents cannot be displayed on the screen at a time, the system searches nearby devices and if it finds them, the system displays one screen using a plurality of devices coordinating with one another.
Furthermore, the system described in JP-A 2004-318207(KOKAI) automatically collects information on screen creating elements and screen display elements included in a plurality of nearby devices and automatically forms a processing configuration which combines the screen creating elements and screen display elements in user specified order or in a preset priority order.
While a multi-monitor function of Windows presupposes a relatively fixed way of combination between a PC and displays such as a desktop, using the methods described in the above described two patents facilitates the formation of a multi-display environment when a multi-display is necessary at a time of using a portable device or the like.
As described above, the systems described in JP-A 2003-61140(KOKAI) and JP-A 2004-318207(KOKAI) allow a multi-display environment to be easily used even when the apparatus configuration changes frequently such as when portable apparatuses are used. However, when a display is started up in a multi-display environment, which display is started up greatly affects the convenience of the user. For example, in a conventional Windows multi-monitor environment, a completely new window is started up at a primary monitor. When a display to be shown is not a primary monitor, it is necessary to move the window from the primary monitor to a desired display after the window is started up through mouse operation or the like. When the window is started up for the second time, it is started up at the position where it was closed last time, but in an apparatus configuration which changes fast from one moment to another, it would be rather unlikely that the position where the window was closed last time coincides with the desired position.
The system described in JP-A 2003-61140(KOKAI) can display one big screen using a plurality of devices in cooperation, but it does not refer to the position of a window when the window is started up. On the other hand, the system described in JP-A 2004-318207(KOKAI) simplifies or automates the determination of an apparatus configuration but it does not perform control on which apparatus a new window is displayed after the apparatus configuration is determined. In any case, there is a problem that a complicated operation to move the window to a desired display is required to display the window on a desired display when the window is started up.
According to an aspect of the present invention, there is provided with a screen transmission device, comprising:
a first receiver configured to receive usage state information indicating usage states of a plurality of screen display devices by a user from each screen display device;
a second receiver configured to receive a message of requesting start-up of a window from the screen display device out of the screen display devices;
a determiner configured to determine a screen display device on which the window is to be started up out of the plurality of screen display devices based on the usage states of each screen display device;
a screen data generator configured to generate screen data of the window; and
a transmitter configured to transmit the screen data of the window to the screen display device determined by the determiner.
According to an aspect of the present invention, there is provided with a screen display method, comprising:
receiving usage state information indicating usage states of a plurality of screen display devices by a user from each screen display device;
receiving a message of requesting start-up of a window from the screen display device out of the screen display devices;
determining a screen display device on which the window is to be started up out of the plurality of screen display devices based on the usage states of each screen display device;
generating screen data of the window; and
transmitting the screen data of the window to the screen display device determined to be started up.
According to an aspect of the present invention, there is provided with a computer readable medium storing a computer program for causing a computer to execute instructions to perform steps of:
receiving usage state information indicating usage states of a plurality of screen display devices by a user from each screen display device;
receiving a message of requesting start-up of a window from the screen display device out of the screen display devices;
determining a screen display device on which the window is to be started up out of the plurality of screen display devices based on the usage states of each screen display device;
generating screen data of the window; and
transmitting the screen data of the window to the screen display device determined to be started up.
In a multi-display environment in which a plurality of displays that can be held by hand are used simultaneously and even in a situation in which the user is changing the hand held displays one after another, this embodiment detects a display on which the user's attention is focused and automatically controls the position of a window to be newly started up so as to be displayed on the display of the highest attention level to thereby alleviate the burden on the user of window arrangement. Hereinafter, this embodiment will be explained in detail.
[Determining Display Destination According to Hand-Held State Information]
The screen transmission device 11 generates screen data for the screen display devices 1 to 3 and transmits them. The screen display devices 1 to 3 display received screen data. The user proceeds with work using the plurality of screen display devices 1 to 3.
The display 21 displays a screen composed by the screen data receiver 22.
The communication controller 23 transmits/receives data to/from a network. The communication controller 23 needs to establish a connection with a screen transmission device which becomes the destination device before sending/receiving data. TCP/IP is generally used as a communication protocol for transmission but it goes without saying that the communication protocol is not limited to this. Furthermore, it is necessary to specify the screen transmission device which becomes the destination device, but the method of specification is not particularly limited and when, for example, TCP/IP is used, the user may specify an IP address or if there is an environment in which a DNS (Domain Name Server) is available, a host name may be specified, the DNS may convert the host name to an IP address and then form a communication channel. The data to be transmitted includes a user operation event which is sent from the user input analyzer 25 and hand-held state information which is sent from the hand-held state detector 27 and the communication controller 23 transmits the user operation event and hand-held state information to the specified screen transmission device. The data to be received includes screen data which is transmitted from the screen transmission device. When the communication controller 23 receives the screen data, it passes the received screen data to the screen data receiver 22.
The input device 24 is the device to allow an input by a user and sends a signal according to an operation input by the user to the user input analyzer 25. The type of the device is not particularly limited and a mouse and touch panel, pen input, keyboard or a combination thereof may be used.
The user input analyzer 25 converts the input signal which is sent from the input device 24 to a user input event on the computer screen which is displayed on the display 21 and passes it to the communication controller 23. The input event may be movement of a pointer, Up/Down of the right/left button of the mouse, Up/Down of a key or the like. The converted event is passed to the communication controller 23 to be transmitted to the screen transmission device.
The pressure-sensitive device 26 is a so-called pressure sensor. The pressure sensor may be disposed, for example, in areas (peripheral part of the display) indicated by reference numerals 27A and 27B in
The hand-held state detector 27 receives the signal from the pressure-sensitive device 26 and judges whether or not the user currently holds the own device by hand. The hand-held state detector 27 is always monitoring the signal from pressure-sensitive device 26 at least while a screen is displayed on the display 21 and sends, when there is a change in the detection result of the hand-held state, the detection result to the screen transmission device through the communication controller 23. Alternatively, it may also store the detection result and transmit the information stored when there is a request from the screen transmission device.
A pressure sensor is used as the sensor to detect the hand-held state in this embodiment, but the sensor is not particularly limited to this. For example, it is possible to use a temperature sensor to detect the hand-held state from the difference in the temperature of the hand when the own device is held by hand and when not held by hand or use an acceleration sensor to detect the hand-held state by distinguishing a state in which the own device is stationary, placed on a desk or the like from a state in which the own device is held by hand and moving around. It is also possible to mount a camera in each screen display device and judge that the device is held by hand from an image taken by the camera if the user is included in the image. Alternatively, it is also possible to use a combination of these methods.
The screen data receiver 22 processes the screen data sent from the communication controller 23. If the received screen data is compressed data, it expands the data and overwrites screen data in an appropriate position of a frame buffer (not shown) which stores the whole screen in the screen data receiver 22. The screen data stored in the frame buffer is sent to the display 21 and displayed.
The screen data generator 31 generates a so-called computer screen. It generally provides a frame buffer in a size which corresponds to the resolution of the screen for storing the latest screen to be displayed. When screens are generated for a plurality of displays, it provides frame buffers for each display and generates screens for each display. The generation of a screen itself is performed through coordination between an application, OS and display driver or the like and the image generated is written into the frame buffer. When informed of a connection of a new screen display device from the display device manager 37, the screen data generator 31 provides a frame buffer for the screen display device. That is, a frame buffer is provided for each screen display device connected. When shown on a display (not shown) of the own device, the written contents of the frame buffer are sent to the display. Upon receiving a user operation event from the user operation event receiver 35, the screen data generator 31 generates a screen (event screen data) according to the type of the event. For example, when the event is a pointer movement event, a pointer is redrawn at a position to which the pointer is moved. It also receives information on which display of the plurality of displays the user's attention is focused most from the user attention level determiner 36. When starting up to generate the screen of a new window to be started up, the screen data generator 31 draws the window on the screen of the display on which the user's attention is focused most.
The transmission screen data acquirer 32 acquires the position in the screen to be updated and drawing data. As the acquisition method, it may be possible to receive all update information from the screen data generator 31 and use the information as is. Alternatively, it may be possible to acquire the latest screen from the screen data generator 31 periodically, compare it with the screen acquired last time and thereby extract the updated part. In addition, a method of hooking drawing-related events such as screen updates used in the system, thereby monitoring which parts are updated and acquiring only the drawing data of the updated parts from the frame buffer at the screen data generator 31 can also be considered. The acquired screen data is sent to the screen data transmitter 33.
The screen data transmitter 33 transmits the update screen data sent from the transmission screen data acquirer 32 to the screen display device through the communication controller 34. The screen data transmitter 33 transmits the screen update position and the drawing data as one set of data. If not compressed, the data size of the drawing data becomes large, and therefore the drawing data may be compressed before being transmitted.
The communication controller 34 transmits data to a network or receives data from a network. The data is transmitted/received to/from the screen display device which becomes the destination device. Prior to a communication, the communication controller 34 receives a request for establishment of a communication channel from the screen display device and a communication channel is established. The communication controller 34 transmits the data sent from the screen data transmitter 33 to the screen display device whose communication channel has been established. The types of reception data include a user operation event and hand-held state information from the screen display device and these are sent to the user operation event processor 35 and the user attention level determiner 36 respectively.
The display device manager 37 manages the information on the screen display device being connected. When informed from the communication controller 34 that a connection with the screen display device has been established and passed user information received through the connection, the display device manager 37 carries out authentication. After succeeding in the authentication, the display device management 37 adds and registers the screen display device which has been newly connected to the display device list and informs the addition of the new screen display device to the screen data generator 31.
The user operation event processor 35 extracts event types such as movement of the pointer and Up/Down of buttons and auxiliary data according to each event (coordinates of the place to which the cursor movement event moves or the like) and sends it to the screen data generator 31.
The user attention level determiner 36 determines the display on which the user's attention is focused most out of the plurality of screen transmission devices used. As the method of determining the attention level, a method of regarding a display which is held by hand most recently as the display of the highest attention based on the hand-held state information received from the screen transmission device may be considered. The user attention level determiner 36 is equivalent, for example, to determiner.
It is also possible to use semi-automatic control as the control of the display position of a window to be newly started up instead of fully-automatic control. For example, it is possible to add such a menu as “To display held by hand” to a menu at the start-up of a new window or a menu of a window being displayed and surely display, when the user selects this menu “To display held by hand”, a window on the display held by hand.
An example of the operation sequence when using the screen display device and the screen transmission device of this embodiment will be described. First, a sequence up to the completion of a connection between the screen display device and the screen transmission device will be described.
An example of a specific usage scene is shown. For example, suppose a list of documents to be checked is displayed on a screen display device A. Here, the user holds a screen display device B by hand and opens a document to be checked by the screen display device A. Then, the document is opened by the screen display device B which is held by hand without any selection as to, for example, from which screen display device the document is opened. Furthermore, when the user wants to see another second document while keeping the document just opened (first document) as is, the user puts down the screen display device B once, holds another screen display device C by hand and then opens the second document using the screen display device A. Then, the second document is automatically displayed on the screen display device C. It is also possible to easily check the second document while comparing it with the first document.
[Determining Display Destination According to Access Frequency]
A second embodiment will describe a method of using access by the user as the method for determining the degree of the user's attention to determine the location where an application start-up screen is displayed. Since the configurations of the screen transmission device and the screen display device are the same as those of the first embodiment, only differences in processing of each block will be explained. Upon receiving a user event, a user operation event processor 35 of the screen transmission device informs from which screen display device it received a user event in addition to the processing described in the first embodiment to a display device manager 37 as user event reception information. Upon receiving the user event reception information, the display device manager 37 writes information indicating that there was latest user access in an entry of a screen display device which sent the information in a display device list.
Furthermore, as another method for determining the user attention level, a method of determining the user attention level based on a user access history may also be considered. This method is the same as the method described above in that the display device manager 37 is informed of from which screen display device the user operation event processor 35 received a user event, but the display device manager 37 which has received the information does not store the information on the screen display device which was accessed by the user most recently, but counts the number of times each screen display device is accessed for a certain period of time. That is, the display device manager 37 increments the count by +1 every time a user event is sent from the screen display device. When the user attention level determiner 36 determines a screen display device on which a new application is displayed, it selects a device with the highest number of times of access.
Furthermore, as another method for determining the user attention level, a method of determining the user attention level based on the screen update frequency (screen generation frequency) of each screen display device may also be considered. When the screen data generator 31 generates an update screen for a certain screen display device, it informs that to the display device manager 37. The display device manager 37 which has received the information increments the screen update count of each screen display device by +1 every time it receives information. When the user attention level determiner 36 determines a screen display device on which a new application is displayed, it selects a device with the highest number of screen updates.
A method of using hand-held state information together with the method described in this embodiment may also be considered. For example, when there is a display held by hand, the user attention level determiner selects the display held by hand as the display from which a new application should be started up assuming that the display has the highest attention level, but when there is no display held by hand, the attention level may be judged using the above described method.
[Concurrent Use of Write-Inhibit Attribute]
A third embodiment will describe a method using a write-inhibit attribute which indicates that a screen display device is write-inhibited as the method of determining a device on which an application start-up screen is displayed. Since the configurations of the screen transmission device and the screen display device are the same as those in the first embodiment, only differences in processing in each block will be explained.
When the user operation event processor 35 receives the user operation event indicating that the write-inhibit button was pressed, the user operation event processor 35 informs this to the display device manager 37. The display device manager 37 writes the information that the device is write-inhibited in an entry of the screen display device in the display device list.
In this way, when the user wants to have an application screen displayed on the screen display device held by hand, the user may start up an application without pressing the write-inhibit button. Furthermore, when the user wants to have a new application screen displayed on a different screen display device which is not held by hand, the user presses the button of the hand held screen display device when the application is started up, and can thereby have a new application screen displayed on the other screen display device.
Number | Date | Country | Kind |
---|---|---|---|
2006-91826 | Mar 2006 | JP | national |
Number | Name | Date | Kind |
---|---|---|---|
6509911 | Shimotono | Jan 2003 | B1 |
6522309 | Weber | Feb 2003 | B1 |
20030197659 | Arai | Oct 2003 | A1 |
20030231143 | Nakamura et al. | Dec 2003 | A1 |
20040125044 | Suzuki | Jul 2004 | A1 |
20050093868 | Hinckley | May 2005 | A1 |
20050253775 | Stewart | Nov 2005 | A1 |
20060001593 | Baudisch | Jan 2006 | A1 |
Number | Date | Country |
---|---|---|
2003-061140 | Feb 2003 | JP |
2003-280783 | Feb 2003 | JP |
2004-023317 | Jan 2004 | JP |
2004-318207 | Nov 2004 | JP |
Number | Date | Country | |
---|---|---|---|
20070234222 A1 | Oct 2007 | US |