Patent Application
20070192727
Data access and retrieval has always been an important aspect of computers. Different data retrieval and data display models have been proposed over the years, but most system designers return to one of three methods due to their simplicity, ease of use, and user comprehensible models. These three models include the desktop model, the list based model, and the hierarchical list model.
The desktop model popularized by Apple® with its Macintosh® computers is used to display computer operating system data in a virtual desktop. On a computer screen is shown an image of a desktop with files, applications, a trashcan, and so forth. Access to files is achieved by selecting icons and opening files/folders associated therewith to reveal either further files or to access the file so opened. Though the model is convenient, it is often difficult to use due to system level constraints. For example, Windows® a popular operating system provided by Microsoft® has limitations on file name length and, as such, is sometimes unable to store files sufficiently deeply within nested folders to truly reflect the desktop based model. Further, since some systems are more limited than others, the model when implemented results in some limitations on portability. For many applications and for application execution, the desktop model is often poor.
Also, though the desktop model is well suited to providing user references for many different functions, it is poorly suited for organizing large volumes of data since it has no inherent organizational structure other than one set by a user. Thus, similar to actual physical desktops, some virtual desktops are neat and organized while others are messy and disorganized. Thus, for data organization and retrieval, the virtual desktop model is often neutral—neither enhancing nor diminishing a user's organizational skills.
The list-based model is employed in all aspects of daily life. Music organization programs, display music identifiers such as titles and artists in a list that is sortable and searchable based on many different criteria. Typically, sort criteria are displayed as column headers allowing for easy searching based on the column headers. Many applications support more varied search criteria and search definition.
Another example of list based data display is Internet search engines, which typically show a list of results for a provided search query. The results are then selectable linking the user to a World Wide Web Site relating to the listed result. Unfortunately, with the wide adoption of the World Wide Web and with significant attempts to get around search engine technology—to “fool” the search engines—it is often difficult to significantly reduce a search space given a particular query. For example, the search term “fingerprint” returns a significant number of results for biometric based fingerprinting similar to that used by police and a significant number of results for genetic fingerprinting using DNA. These results are distinct one from another.
The hierarchical list is similar to the list-based model but for each element within a higher-level list, there exist further sub-items at a lower level. Thus, a first set of folders allows for selection of a folder having within it a set of subfolders, etc. This allows for effective organization of listed data. In the above noted music list program example, classical music can be stored in a separate sub list from country music.
Unfortunately, the desktop model, which has been beneficial to the widespread acceptance of personal computers generally, is often insufficient for effective organization of data and applications within organizations, distributed environments and handling large amounts of information. It would be highly advantageous to provide a graphical user interface that is better suited to these users needs and supports both simple navigation and utilization.
An embodiment of the invention teaches user interface comprising: data stored within memory, comprising; first data at least indicative of a three dimensional virtual environment and relating locations within the three dimensional virtual environment, and application data relating to at least one of a plurality of software applications for execution by a processor,
a first input device for receiving user input signals and for providing first control signals to the processor, the first input device for providing data indicative of at least a change in the viewpoint of the user; and, a first display for providing to the user an image generated by one of the plurality of software applications associated with a portion of the three dimensional virtual environment, the portion of the three dimensional virtual environment determined in dependence upon the first control signals and including at least one graphical representation; wherein first data associated with a graphical representation provide corresponding application data to be accessed, said application data retrieved in dependence upon the graphical representation and the location of the graphical representation.
Embodiments of the invention also support a method comprising: providing a computing device comprising a user input port and a display; providing image data to the display, the image data supporting a user interface that provides images corresponding to a three dimensional virtual environment; providing within the three dimensional virtual environment portions generated by an application and other than an operating system thereof, those portions generated by the application other than during user initiated execution thereof and forming a representation of the application for selection for execution thereof; receiving an input signal from a user via the input port, the input signal indicative of user interaction with the portion representative of the application; and executing the application in dependence upon the input signal.
Exemplary embodiments of the invention will now be described in conjunction with the following drawings, in which similar reference numerals designate similar items:
Referring to
Clearly, the prior art GUI represents a simple model for finding those items that are well organized; however, in many cases, the location of specific information is not presented in an intuitive way. Thus, a worker optionally organizes their virtual desktop 101 in a fashion that they can efficiently navigate; unfortunately, not all workers are so organized. Specifically, a worker with a disorganized desktop is likely to have a disorganized virtual desktop 101. As such, many individuals with an “out of sight out of mind” attitude have a desktop that is cluttered and filled with files and folders and applications. Equally the prior art GUI rapidly becomes cluttered and difficult for even an organized and methodical worker active on a large number of documents and applications.
Unlike a true desktop which is within an office, the desktop is much more difficult to organize effectively and intuitively. For example, the background of the desktop, intended to be pleasing to look at, is covered and cluttered by anything on the desktop. Other folders are difficult to intuitively arrange. Further, cleaning of the desktop and folders results in reorganization of all the files and folders, which is often undesirable. Further, one method of organization is different from another and the though the interface is intuitive in some ways, it is not intuitive in many others and, as such, someone else will have difficulty locating files, folders and applications.
Referring to
Also shown is the bottom toolbar 116 which typically displays icons for open applications 115, folders 113 and files 114 alongside other items according to the users settings which may include, but are not limited to, clock, virus protection, network availability etc (which are not shown for clarity). The graphical user interface provides quick and convenient access to applications and files located on the desktop. For files and applications that are less often accessed or that require greater organization, folders are nested one within another to provide for organized storage of data and applications.
In order to overcome this problem, Microsoft® has added a Start® button to the Windows® user interface providing access to applications, settings and to recent documents. This is shown in
Thus, a feature outside of the desktop paradigm is needed to provide convenient functionality of the “intuitive” user interface.
Referring to
As is evident to those of skill in the art, a user interface based on the virtual space provides a two-dimensional view of a portion of the space—a viewport. Navigation tools may allow a user to navigate throughout the space in order to change their current viewport. For example, during typical work, a user sets the viewport to the virtual desktop 202 of their virtual desk 205. In use, the virtual desktop 202 provides much of the functionality of the prior art graphical user interface 101, and mimics their normal actions when working with documents, files and folders on a real physical desk. It is therefore very familiar and simple to use for the user. Thus the virtual desktop 202 supports providing files and folders as well as access to frequently used applications. For example, selecting a blank lined pad of paper triggers the loading of a document generation application. Also, like the prior art virtual desktop 101, the virtual desktop 202 according to the invention supports conventional functions for opening and otherwise manipulating files and folders in a fashion consistent with the prior art.
To access an existing, but currently unopened file, the viewport is redirected to the file cabinet 203. The virtual file cabinet 203 functions in a fashion consistent with a real filing cabinet but with the added functionalities provided by electronic data storage such as search and retrieval, indexing, correlated data, time stamped data, and so forth. In this case the file cabinet is shown with two drawers, 203a and 203b, and may be labeled for ease by the user with their references much like a real physical cabinet. Optionally, as shown a file drawer 203a is shown locked, but it could also be the whole file cabinet 203. If the user within the virtual desktop goes to extract from the locked file drawer 203a then the user is prompted to provide authorization data prior to access thereto. Alternatively, for filing cabinets that are not locked, such as 203b these open files/folders are displayed in a manner predefined by the user.
The virtual telephone 204 supports telephone services using a virtual telephone interface. Alternatively, the virtual telephone 204 supports telephone services using a real telephone supporting a data connection with a computing device that supports the virtual environment. When the user provides an appropriate input signal to the virtual telephone 204 a telephone interactive application is optionally launched. Alternatively, the application is a terminate stay resident application, a constantly available application in execution, or functions of the application are executed from the graphical user interface which is in common execution. This application facilitates the use of telephone functions such as address book look-up, three-way calling, call waiting, message center, and so forth.
The virtual workspace 201 is a virtual three-dimensional environment. The virtual workspace 201 supports commands that allow a user to change the location of and orientation of their viewpoint within the virtual workspace. Thus, while one would expect to use the virtual desktop 202 frequently, in situations where the viewpoint of the user is not directed to the virtual desktop 202, the virtual desktop 202 need not be shown.
Referring to
By adjusting the viewpoint, the virtual desktop 202 is likely to no longer be provided in the field of view. Optionally, the user is provided an icon that automatically, and rapidly, returns their viewpoint to a predetermined location such as the virtual desktop 202. Alternatively, the user returns to a specific viewpoint when a specific input signal is provided. Thus, for example, by touching a monitor for five seconds, the viewpoint is automatically returned to a default position. Further optionally, the user is provided the option of producing an icon that changes the viewpoint to a predetermined location specified by the user.
In order to convey useful information, the printers 214, 215 and 216 in the image 210 are shown having a size that is consistent with their actual size. In addition, those printers, for example 216, that have a multitude of trays for providing paper are shown supporting a corresponding set of trays, 216a to 216c. Optionally, other data regarding the trays, such as paper size and default paper color are provided. The user is then able to inspect the printers and make their choice between the real printers by selecting the image corresponding to the printer. In this way, the user is provided information consistent with the actual printers without having to physically move to the printers to acquire that information.
Similarly, with reference to
The first embodiment of the invention acts to provide information to a user in a fashion that is consistent with how a user would normally acquire corresponding information in the physical world. In addition, the user is provided an interactive system that is functionally consistent with a real system. This consistency reduces the amount of training necessary for most users to learn a new system. Further still, the embodiments of the invention support the use of relatively complex services, such as telephone services, via the virtual environment with minimal training in the use of those services. Finally the addition of supportive hardware such as eye-tracking or head-tracking software would further add to the consistency with a real world environment as the system could react to natural behavior of the user rather than requiring keyboard, mouse or other pointer device inputs to control the viewpoint and functions.
In order, to support the functionality of the virtual environment, supporting devices optionally provide interface data useable by a computing device generating the virtual environment. Thus, for example, when a new printer is installed in the working environment and said printer provides suitable data communication with the computing device that provides the virtual environment, the virtual environment is optionally updated to indicate a virtual representation of the new device. Similarly, when a device is no longer in a state of suitable data communication, it is no longer represented within the virtual environment. Alternatively, such devices are shown as non-functional in the virtual environment when suitable data communication is not available.
Expanding upon this
Certain virtual items located outside the virtual office of the user do not support interaction with the user. In this way, large and complex virtual workspaces with many distinct users are optionally supported. Considering the calendar the user can now advantageously provide notes to the calendar including reminders, have documents distributed for a meeting linked to the meeting notice on the calendar so that retrieving them is quick, and also automatically have these updated as different releases of documents, such as the meeting agenda, are distributed. In addition, the calendar is optionally used to open a personal scheduling program that supports substantially more detail than a conventional calendar would. Some events are optionally provided to the calendar automatically. Further, by placing of files in front of calendar events, etc. it is possible to organize data in a temporal fashion as well as in a special fashion.
Optionally, a person who has a personal digital assistant (PDA) or sophisticated email pager and cellular telephone has a virtual PDA shown in their virtual workspace 201. If the user has the PDA undocked from its docking station then the user can have the system maintain an icon for the PDA in any viewport the user subsequently assesses in a predetermined position, see for example 224 in
By activating the virtual PDA, real PDA functions are controlled. Thus, the user easily transfers data, updates schedules etc. between the PDA and the virtual workspace 201. In addition communication between the PDA and the users computer 209 is optionally controlled, for example by providing an input signal within the virtual environment. Consider for example, the virtual PDA is optionally brought into close proximity with the virtual calendar. In response to this proximity, a predetermined function is carried out supporting, for example, the synchronization of a scheduling system in the PDA with a scheduling program of the computer. Upon completion of this task the PDA icon returns to a predetermined location.
In the case of a PDA, it is possible to advantageously add additional functions such as denoting where a colleagues' PDA is, for example using wireless assisted GPS. Hence, when needing to quickly meet a colleague to discuss something the user can locate them rather than hunting them randomly within the working environment. Equally the system could determine when the real PDA is determined to be in a predetermined location within, for example, an office, or outside of the office, and hence indicate that said colleague is out of the office automatically when the user selects this colleague on the virtual telephone.
Referring to
Additionally the user has added shortcuts as follows:
virtual computer 305, which for example could be a server that the user is responsible for maintaining, a remote computer within a laboratory performing tests under the users direction, or a variety of other advantageous links to additional computers;
second virtual calendar 308, which for example could be the merged vacation records for the users team, bookings for a conference room the user is responsible for, or a variety of other advantageous links to additional calendars;
a virtual printer 309, which for example might be the printer in the printing room of a design company, a printer at a colleagues desk in a remote facility, or a range of other advantageous uses of a link to a remote computer;
a second virtual printer, which for example might be the printer of the central sales office wherein the user can print orders from his satellite office;
a second virtual filing cabinet 312, which for example may be central personnel records, accounting or other such centralized records, which would normally be managed through complicated links to multiple servers and/or directories.
In this manner the user optionally customizes their virtual desktop to reflect their actual operating requirements. It would be evident therefore to one skilled in the art that this embodiment, and others provide for an increased efficiency of an employee's time and resources, and provides for organization of a users environment at many levels, rather than the single uniform structure of a conventional desktop operating system, GUI, and software environment.
Referring to
Thus, for example, a user stores a set of icons 402a to 402c on the virtual desktop of the second monitor. The icons 402a to 402c representing viewpoints of three different locations within a physical building which is denoted in the virtual desktop image by the building 403. A first icon 402a corresponds to a predetermined viewpoint of the virtual environment. By interacting with the icon 402a, an image associated with the viewpoint is shown in first monitor. Clearly, the first monitor is optionally used for other functions. For example, the first icon 402a links the user to a viewport associated with their own desktop allowing them to perform their normal desktop actions.
The second icon 402b corresponds to a predetermined viewport of a manufacturing environment. In selecting this icon the viewport on the first monitor adjusts to reflect the selected viewport. This for example might allow the user to adjust the operation of manufacturing machines, check the status of manufacturing schedules, and look at inventory or a variety of other advantageous actions from their desk. In the embodiment considered here the user is able to manipulate the viewpoint of either monitor. In addition, the viewpoint in both monitors is optionally changed in response to a same input signal.
Finally, the third icon 402c corresponds to a predetermined viewport of the corporate headquarters. In selecting this icon the viewport on the first monitor adjusts to reflect the selected viewport. This for example might allow the user to work on centralized manufacturing, quality, and financial records as opposed to their personal localized files, which are managed through their personal virtual desktop access through icon 402a.
Within the embodiment as described it is possible to adjust the users ability to manipulate their virtual desktops as accessed through the different icons 402a, 402b and 402c. This can provide a different approach to security as for example a clerk might only be able to access a virtual desktop, which is fixed and predetermined. A supervisor may be able to adjust to some degree the virtual desktop, for example to reflect their teams locations, operations and even accessing additional resources above and beyond those available to the clerk, for example a printer or external networks.
It may also be beneficial in some instances that the viewports 402a to 402c are not predetermined but can be changed according to the user selecting a room within the image presented of the building 403. Hence, the image of the building 403 might be a floor plan and selecting a room causes a virtual desktop icon to appear on the second monitor, which can then be selected and worked with on the first monitor. In this manner for example a user can work on a presentation, then select a meeting room, schedule a meeting, invite attendees, and finally store the presentation on the projection system within the meeting room.
It may also be beneficial in some instances and embodiments of the invention for certain virtual elements to remain consistent within the computer interface and hence denote some virtual elements are designated as being fixed, whereas others are not fixed. Thus, it is possible in other embodiments of the invention to allocate additional properties to elements of the virtual desktop providing for unforeseen advantages in the efficiency and operational effectiveness of the users accessing the virtual desktops.
Hence, for example, one of the shared printers that has a physical presence at the end of the hallway, as described previously in
Clearly, there are other virtual items within the virtual environment, which are mobile, and permanently moveable by any user. Consider, for example, in a work environment where any worker is available to do piece work, and a set of work orders are provided on a virtual bulletin board. A virtual presence of a worker is able to review the work order and, should the worker decide to do work of one specific work order transfer the work order off the virtual bulletin board to say a virtual inbox for that worker. The workers supervisor being able to see both the virtual bulletin board and the inboxes of all workers can therefore manage the workers and work flow within the work environment.
In an alternative embodiment of the invention, the interaction of a user within a virtual environment is the manipulation of real devices, as well as the manipulation of documents, text etc. For example, a user optionally integrates the lighting system of their office within the virtual workspace displayed upon their selection of icon 402a in the third embodiment of the invention. The lighting system is designed to be responsive to input signals provided by a designated computing device. Thus, a user interacting with their virtual desktop and toggling a switch within the virtual environment causes the appropriate signal to be communicated to the designated computing device handling lighting and therein turn on or off their office lights. It would be appreciated to one skilled in the art that alternate embodiments are possible wherein the virtual desktop adjusts to reflect the users actions, and in this case for example brighten the desktop to respond to lights turning on or “turn on a virtual light bulb”. In this manner the virtual desktop can change in visual presentation to the user allowing the current environment to be accurately reflected.
Clearly, other embodiments are easily envisioned in which a user manipulates a virtual item to achieve a real response. For example, in a manufacturing environment a set of machines perform a set of predetermined tasks. A virtual environment associated with the manufacturing environment is provided. A real inspection station inspects manufactured items for defects. A user is able to provide a virtual input signal to an inspection device to provide a specific view of an item in inspection. If the item undergoes a cleaning operation prior to inspection but the inspected item is not clean, the user optionally changes their viewpoint to a cleaning station. The user is then provided image data from the cleaning station. The user optionally changes parameters associated with the cleaning station by providing input signals to the virtual environment. These changes result in corresponding changes to the cleaning process. Clearly, in such an embodiment of the invention it would be beneficial to provide a relatively large number of monitors such that plurality of different stations are optionally viewed at a given time. Alternatively, a given monitor supports a plurality of different viewports.
In another aspect of the invention, the user wishes to extract a document from their filing cabinet, which has two drawers. Much like the real physical world the virtual desktop can display the names of the drawers, say General and Confidential. The user clicks on the filing cabinet drawer marked Confidential, which is shown as locked. Upon providing the correct verification of the users identity, which can be from a variety of means including and not limited to the direct entry of security data, connection of a USB security dongle with automatic verification, and even biometric data, the drawer opens to display a series of files. In this case the files are visually indicative of their contents, which optionally include degree of sensitivity of materials stored, number of records within the file, memory space consumed or many other predetermined settings.
The user selects the appropriate file, which opens, and now in this embodiment and unlike an existing desktop solution, the system presents the records firstly according to predetermined preferences. These for example could be to present all graphical images as if in a picture album but now with each page having additional notations such as originator, date, contents etc. For written documents the presentation could be as a library with filenames displayed on the spines, or a folder with the title pages for leafing through much like a photo album. The system could allow the user to shift views should they experience difficulty or even call up search tools. As such these embodiments leverage the human attributes of sight and memory in manners closest to our real world experiences and normal behavioral patterns, rather than those determined by a desktop software company with long lists of similar names, sorted alphabetically or historically rather than contextually etc.
Though the term peripheral device is used herein with relation to computer peripherals such as printers, it is also envisaged that external sensors, monitors, or other peripheral devices are included within the scope of the term peripheral.
Numerous other embodiments may be envisioned without departing from the spirit and scope of the invention.
This application claims benefit of U.S. Provisional Patent Application No. 60/762,128 filed Jan. 26, 2006 and No. 60/762,514 filed Jan. 27, 2006, the entire contents of which are incorporated herein by reference. The present invention relates to User Interfaces and more particularly to a method and system for providing a user interface operable for applications including operating systems.
| Number | Date | Country | |
|---|---|---|---|
| 60762128 | Jan 2006 | US | |
| 60762514 | Jan 2006 | US |
