1. Field of the Invention
The present invention relates generally to methods for defining relationships between, and operating graphical objects.
2. Description of Related Art
A user interface is a configuration of widgets conjoined with logic managing how each relates to data and interacts with other widgets for the purpose of facilitating dataflow, workflow, and ultimately program flow. The purpose of a user interface (“UI”) is to provide the user with data visualization, selection capabilities, and command mechanisms. The combination of these functions and how they interrelate facilitates workflow, dataflow, and program control flow inherent to an application. User interface widgets facilitate data representation by rendering data, editing data, and/or providing lists of data items. Many widgets are selection sensitive and provide the capability to select sets of data items that can then be operated on when commands are issued to do so. These commands originate from command sensitive widgets. In conventional usage, user interface objectives and requirements may not be fully addressable because of the difficulty associated with meeting these objectives and requirements using source code.
In a conventional UI paradigm, user interface widgets are generally divided into four types:
Generally speaking, most widgets have some amount of visualization associated with them. For the purposes of categorizing widgets, a widget's primary function determines the widget type. Table 1 lists some commonly used widgets and their primary function:
A conventional UI paradigm groups all widgets into the “Command” category so that each widget issues commands and/or notifications indicating what the widget is doing. It is then the responsibility of the application developer to manage these notifications in order to create the intended widget behavior as described by the categories above. The conventional paradigm additionally includes the concept of aggregate widgets including, for example, tab controls or property sheets. This concept embodies an aggregation of widget primitives and would be better described as a widget configuration rather than a widget itself.
A commonly used conventional paradigm for selection management depends on having a distributed selection context for the UI where each widget internally stores and maintains the selection state relevant to that widget. This paradigm is used primarily because selection information is needed during widget rendering and also when interacting with a user when it is desirable to have the selection information at hand within the widget to facilitate widget rendering and user interactions.
Furthermore, each widget has a well-defined purpose and can only display certain types of data resulting in a rigid interface for transferring data to/from the widget as well as manipulation of the internal selection state. The standard set of widgets generally only support primitive data types such as a string, a number, etc. or collections thereof. A widget typically has a primary purpose of displaying data in a form understandable by a software user. Thus, each widget is typically connected in some way to data storage for the purpose of displaying that data, which can involve writing to data storage as well as reading (in the case of editing data).
Data storage is data having unknown structure for which the user interface is being constructed and the structure is at least unknown to a widget toolkit builder. Since only the simplest data storage is directly compatible with conventional widget interfaces, a mediator is needed to provide the “translation” and data path between widgets and their data storage. This data flow process between widgets and data storage is known as data binding. To facilitate software workflow and dataflow, mediators perform a variety of tasks that can be divided into four groups:
1. Data transfer facilitating a data path between widgets and their data sources in addition to querying widget selection states in order to decide what data to provide or otherwise access at any moment.
2. Widget event handling related to data and/or to selection context and encompassing management of widget selection state and interdependent states of widgets; this may also include providing new data to widgets when a change in selection state requires different data to be displayed in any widget.
3. Widget event handling related to command and control and encompassing actions directly related to performance of operations on data by widgets based on the current selection context. Typically, this involves receiving an event and then having to query widget selection states in order to determine how to best handle that event.
4. Miscellaneous event handling and control flow encompassing program flow, configuration changes, data management, layout resizing, and other miscellaneous tasks most likely related to workflow, dataflow, and control flow.
In particular,
The main difficulty regarding the UI mediator, in terms of software development, lies in having to implement the logic driving widget selection-context management and bound data. This is primarily rooted in the conventional paradigm having a distributed selection context requiring the mediator to query widget selection states for nearly every logic step.
Certain embodiments of the present invention comprise systems and methods address issues associated with conventional user interface paradigms that have a distributed selection context having selection states spread across user interface elements (widgets). Certain embodiments isolate the binding between widgets and data storage using a centralized selection context paradigm for a majority of data items represented within a user interface. This selection context is fully accessible by methods that access data while considering the current selection context. The centralized selection context simplifies event-handling logic by having selection context readily available thus removing the need to query individual widgets for their state. In addition, the user interface workflow graph can be derived from the selection set definitions.
Embodiments of the present invention will now be described in detail with reference to the drawings, which are provided as illustrative examples so as to enable those skilled in the art to practice the invention. Notably, the figures and examples below are not meant to limit the scope of the present invention to a single embodiment, but other embodiments are possible by way of interchange of some or all of the described or illustrated elements.
Certain embodiments of the invention employ methods and systems that facilitate the creation and operation of a user interface consistent with user objectives and requirements. Data visualization, selection capabilities, command mechanisms, methods and systems are provided to enable improved workflow, dataflow, and program control flow inherent in an application.
As used herein, “user interface” or “UI” means the human-computer interface allowing the computer user to interact with computer software. As used herein, “widget” or “user interface element” means a visual object representing some sort of data item the user can interact with on the computer monitor through the use of a keyboard, mouse, or some other means. Well known user interface widgets are: button, listbox, combobox, tree, menu, scrollbar, etc. As used herein, “data source/storage” means a container for the data displayed by a widget usually manifesting as memory during program runtime but could be resident in permanent storage. As used herein, “data binding” encompasses the implementation of the data flow between a widget and the data source or sources it represents. As used herein, “selection” or “selection state” means the dynamic state of chosen sets or subsets of data for visual depiction or upon which to perform operations. As used herein, “selection context” means the collection of multiple selection states. As used herein, “workflow” describes how a user traverses a user interface in terms of which widgets are interacted with and when, how often, the screen space traversed between widgets, and how this changes throughout a software session. As used herein, “Mediator” is a set of logic rules determining how to respond to, handle events or actions, and routing data flow between a variety of otherwise incompatible software components, such as widgets and data sources. As used herein, “Widget Toolkit Builder” means the provider or developer of a software development kit (SDK) supporting a generic set of widgets that an application developer can use to build software products/applications. As used herein, an “Application developer” is a software developer responsible for designing and assembling a software product using available tools in order to create a particular workflow and dataflow in order to solve a particular problem or provide a particular service. As used herein, the terms “implement” or “implementation” are used primarily to describe the process of converting a concept, behavior, or feature into program source code.
A user interface is a configuration of widgets conjoined with logic managing how each relates to data and interacts with other widgets for the purpose of facilitating dataflow, workflow, and ultimately program flow.
Thus, when a widget event occurs 212, the mediator 21 typically queries the widget selection context 213 and retrieves the selection from the widgets 214 in order to appropriately handle the event 215. To handle the event, the mediator may determine if the event modifies data storage in any way 216. If modification is indicated 217, then data can be read or written to data storage 218. If modification is not indicated 219, the mediator then may determine if the selection context has changed 220 in such as way that dependent widgets require new data. Otherwise 221, control may be returned to the executive 208. Where widgets need new data 222, the mediator can again fetch data 223 from data storage 224 and subsequently process and provide the data 225 to the widgets. The widgets may then store new data 226, including overwriting or modifying existing data. Finally, the mediator may update the selection context for widgets that have changed 227 and the widgets subsequently store the new selection. The control may be returned to the executive 208.
More specifically,
Table 2 categorizes certain user interface toolkits and their corresponding primary data binding methods:
Although other toolkits are commercially available, most are derivatives of those listed in Table 2. For example, MFC is based on Win32 and Cocoa is based on Carbon, thereby sharing the same data binding and selection paradigms. Additionally, toolkits for languages such as Java are founded on conventional data binding and distributed selection principles. Thus, the particular language of implementation is irrelevant.
It will be appreciated that some amount of direct data binding can be implemented in any widget toolkit. However, most widget toolkits are not designed to use direct binding, with the exception of QT. Additionally, it will be appreciated that all widget toolkits in Table 2 utilize a distributed selection context: i.e., every widget contains its own selection state internally and the mediator must explicitly query the state when making decisions.
In certain embodiments, a data binding paradigm includes centralizing user interface selection context. This centralization permits the selection context to be fully accessible to the widgets, data storage, and mediator. Furthermore, the use of direct data binding yields a powerful, yet easy to use, data binding system, as demonstrated in
In certain embodiments, the mediator can be removed from a data-binding loop provided in conventional paradigms. Such removal is facilitated by permitting the selection context to be freely queried by data storage and allowing free updates and querying by widgets without relying on the mediator.
In certain embodiments then, widgets can communicate directly with the selection context in order to update selection changes resulting from user interaction and can furthermore query the selection context on their own prior to rendering to insure they render the proper state. A data-access interface can be primarily based on the required interface for each particular widget because each widget is typically limited to rendering finite data types. When the widget needs to be filled with data, it typically inquires across the data-access interface to the data storage, which in turn queries the selection context in order to determine the appropriate data to give to the widget.
The mediator may remain involved in creation, destruction, and other management tasks unrelated to data transfer. The mediator can still access the widgets, data storage, and central selection context as required but it is not necessary that the mediator participates in data binding at the same level as it is in conventional paradigms.
Referring now to
In
In certain embodiments, a simplified mediator is provided for use with a centralized selection context. The simplified mediator can be obtained by reducing complex logic required to bind widgets to their data because this process has essentially been split and moved into the widgets and data sources. Additionally, the centralized selection context need only manage selection state and is simple in design. It will be appreciated that, under this paradigm, widgets and data sources are required to provide additional functionality to substitute functionality that was eliminated from the mediator.
In one example, additional widget functionality includes:
In certain embodiments, increases in widget complexity can be mitigated by the well-defined role of each widget. Thus, the finite amount of logic needed for each widget can be managed by the toolkit builder rather than by the application developer, thereby greatly simplifying application development processes. The complexity of data sources slightly increases under this model since the developer must provide an interface through which the widget can communicate with it. Although such logic previously existed in the mediator, the relocation of data gathering to the data source (i.e., the process of preparing data for transferring to or processing received data from the widget) can greatly simplify the gathering process because the data source itself has full knowledge of the potentially complex data structure it represents. Thus, the data source can easily determine an order to data based on context and the structure of data objects without needing to interrogate data sources in order to access their internal data structures.
The conventional paradigm stores selection context for each widget inside the widget itself. When the widget subsequently changes selection state, the mediator must detect this change and provide correct data for the new context to the dependent widgets (see the flowchart demonstrating this process in
The match with flat data structures arises because only one level of selection indirection exists. In the example, the Person object itself. For a bookshelf, multiple levels of selection must be traversed in order to arrive at the page text. Traversing selection levels may represent the major difficulty in implementing conventional user interfaces since there is no supporting mechanism to easily keep track of the multiple selection levels as well as providing this information to the components that need it. The centralized selection context allows this information to be readily available making this process much easier to implement. The centralized selection context is (usually) comprised of several selection sets. Each selection set represents a group of similar data items that are displayed by the user interface. Selection sets are typically a result of the data structure being represented by the user interface.
Pseudo code for the data structure of the bookshelf example is:
The same data structures in a C++ implementation are given by:
In this case, four levels of selection are traversed to arrive at the page text: house, bookshelf, book, and the page itself.
For the purposes of description, the following terms will be used in order to implement a centralized-selection user interface paradigm:
Each of the operators DATUM, SELECTIONSET, and COMMAND are a function that converts the name into the unique identifier. It can also be assumed that passing the same name to different operators results in different identifiers such that DATUM(name) does not equal COMMAND(name). Additionally, there exist declaration versions of each of these: DATUM_DECLARATION, SELECTIONSET_DECLARATION, and COMMAND_DECLARATION. These are used to declare the entity itself for use within a program.
The list widgets in this example simply display a list of strings from which the user selects in order to modify the selection context. The data transfer in this case can be implemented using the following interface:
In which case, upon return of the call to GetSelectionItems( ), vItems is filled with the list of strings to be displayed. Additionally, a handle can be associated with each string and used to represent that particular string datum if the user selects it in the widget.
The preview widget displays text in this example and the data transfer for this widget can be implemented using the following interface:
In which case, upon return of the calls to Set( ) or Get( ), data is either written or read to data storage. The data in this case is transferred using the parameter “data.”
Subsequently, a list widget can be bound to a particular selection set and data storage in the following manner:
Thus, when a list widget needs to till itself with data, it need only call GetSelectionItems( ) on the bound data access pointer m_pAccess. To make this call, it passes the bound selection context as well as the selection set it is querying. It can then use the resulting list to display the data.
The preview widget can be bound to a datum and data storage in the following manner:
In this example, since the preview widget is read-only, only the Get( ) method (defined by the IDataAccess interface) is used by the preview widget.
Using these constructs, the bookshelf data structures, which constitute the data storage in this example, can be supplemented in the following way to apply the centralized selection paradigm to this example:
In certain embodiments of the invention, a mediator may require that four selection sets be created to represent the four selection levels in this data structure in order to initialize the centralized selection context. In C++/pseudo-code this is:
The create method creates the selection set passed as the first parameter and also establishes a dependency on the second parameter if one exists. This manages invalidation of dependent selection sets when a parent changes state, for example, the selected bookshelf must be invalidated when the selected house changes since the bookshelf is no longer in context.
The widgets can be bound to the selection context and data storage in the following manner:
At this point, the selection set dependency has been established as well as how widgets are bound to it and the data storage. These relationships are shown in
A key to the centralized selection context lies in the method by which data is accessed once these relationships are established. This can be appreciated in the implementation of the Get( ), Set( ), and GetSelectionItems( ) methods. For example, the GetSelectionItems( ) method for AvailableHouses is given by the following:
The first part of the method checks to see if the selection set being queried is one that this level in the data structure maintains. If so, items can be filled with appropriate data: in the example, homeowner names. Otherwise, the query may be passed on to the nested house data that is in context. The remaining implementations of this method for the other data objects are similar. The base case is in the Book in that if the selection set passed is not Page, then the method simply returns false since it has nowhere to pass the query.
Subsequently, the Get( ) and Set( ) defined by the IDataAccess interface can be similarly constructed in regard to how they compare the passed datum with those that particular object handles. If the object can handle that datum, it does so. If not, it passes the query on to nested data item that is in context. This sort of data access handling allows widgets to be bound to the topmost data object and is never invasive to the data hiding mechanisms employed in most aggregate types.
With reference also to
The workflow visualizations are shown in
Easy-to-use user interface layouts follow cultural constructs that have been in place for some time. The construct that applies in this case is that of left-to-right (or right-to-left depending upon the cultural language) and usually top-to-bottom organization of information. This ordering establishes a natural progression a person follows in order to arrive at the desired result.
This information can be used for a variety of things such as i) automatic widget placement based on workflow; ii) layout grading or evaluation based on workflow complexity as appropriate for a given culture; iii) visualization of the workflow; and iv) tracking to identify which user interface components including, for example, widgets with which a user spends their time interacting.
Certain embodiments provide systems and methods for file format capture. In one example, the systems and methods are used for capturing the file format within computer software. The invention provides an alternative to external maintenance of format specification detailing the byte-by-byte sequence of data values within a file. In certain embodiments of the invention, the format specification can be captured by the software automatically, thus eliminating the need for external representation and maintenance.
As used herein, a file format specification includes a byte-by-byte description of the sequence of data values stored within a file. A file format specification is the necessary component in data sharing between vendors and for any software system. Nearly every software tool has its own native file format and many are capable of reading or writing non-native file formats typically comprising files created by other software products or vendors. To access non-native file formats, an application developer must implement program code that correctly reads or writes data in the proper sequence. This sequence must be absolutely correct for software applications to be fully compatible with one another. Three methods can be typically used to implement a file format including:
Each of these methods is frequently unavailable due to various reasons. For example, an owner of the intellectual property in source code may not want to distribute their implementation of the file format. This may be because there is proprietary information contained within their particular implementation or simply the owner would have to disclose too much intellectual property for this to be feasible. Additionally, an application developer may not want to use existing source code if it is incompatible with the system to which it would be added. In another example, a format specification may be unavailable due to the difficulty in creating and maintaining the specification for a complex file format. This specification can be a derivative of the source code that implements the file format and requires an individual to analyze line-by-line how the source code reads or writes a file in order to reveal the byte-by-byte specification. This can be an extremely time consuming process for complex file formats and may need to be repeated each time the reading or writing mechanisms are changed. In another example, guessing can be quite effective when performed iteratively. Typically an application developer creates or otherwise obtains a file storing known data and inspects the resulting file byte-by-byte in order to decode how the known data was written into the file. However, for complex file formats, it may be impossible to discover meaning for every byte location in the file format through guessing.
These methods are described graphically in
In this example, the file format originates in source code such that the file format specification is a derivative of the source code. In some cases, the file format specification is originated prior to the first implementation of source code that implements that specification. In this case, the source code is a derivative of the file format specification. However, once these two items exists, it is possible for the source code and specification to become unsynchronized in which case improvements made to either will need to be reflected in the other. Thus, there exists interdependency between the source code and the file format specification requiring maintenance of both.
In certain embodiments of the invention the line-by-line determination of the file format can be modified by automatically capturing file format information directly from the file input/output methods in the source code. In certain embodiments, the file format specification is automatically derived from the source code so that there is only one point of origination and maintenance.
In certain embodiments, write operations in source code are instrumented such that the file format specification can be captured by a “dummy” file capturing object and subsequently processed to produce the file structure specification. This process is illustrated in
Typically, the capturing layer must be capable of capturing the metadata needed to reconstruct the file format specification. This metadata can include data type, size, and description in addition to structural file format elements such as loops, objects, or file sections. Much of the metadata can be captured automatically in the intermediate layer when implemented. However, the basic description explaining exactly what each data item represents in the file format purpose may be lacking. To account for this, an application file I/O method may be instrumented to obtain these descriptions. For example, from the application perspective, assume a write method:
a write method can be provided such as:
Certain embodiments of the invention provide a method for maintaining context for a plurality of widgets, wherein the method comprises maintaining selection state for a plurality of widgets in a central storage; updating the context based on a query from one or more of the plurality of widgets; and selectively providing access to the context to the one or more widgets. In some of these embodiments, the access includes access to the selection state of the corresponding one or more widgets. In some of these embodiments, for each of the one or more widgets, the access is provided to a selection state corresponding to the each widget. In some of these embodiments, the updating includes updating the selection state of the one or more widgets. In some of these embodiments, the selectively providing access includes permitting the one or more widgets to directly query any needed selection without using a mediator. In some of these embodiments, the selectively providing access includes permitting the one or more widgets to directly query any needed selection without using a different widget.
In some of these embodiments, the one or more widgets communicate directly with the selection context. In some of these embodiments, responsive to the directly communicating, the selection context changes based on a user interaction. Some of these embodiments further comprise providing the one or more widgets with data. In some of these embodiments, the providing data includes receiving an inquiry from the one or more widgets, the inquires being communicated through a data-access interface to data storage. In some of these embodiments, the providing data includes transmitting a query to the selection context from the data storage, the transmitted query prompting the return of relevant data to the one or more widgets.
Certain embodiments of the invention provide a method of data binding comprising managing selection context using at least one of a widget and data storage. In some of these embodiments, the selection context is maintained centrally. In some of these embodiments, the managing includes responding to queries from the widget. In some of these embodiments, the queries include queries directed to the entire context. In some of these embodiments, the management includes providing access to the context from the widget. In some of these embodiments, the access is provided to the entire context. In some of these embodiments, the managing includes responding to queries from the data storage. In some of these embodiments, the queries include queries directed to the entire context. In some of these embodiments, the management includes providing access to the context from the data storage. In some of these embodiments, the access is provided to the entire context.
Certain embodiments provide systems and methods for managing a centralized context comprising creating a widget using a mediator and subsequently connecting the widget to an associated data storage. Some of these embodiments further comprise creating a central selection context. Certain of these embodiments further comprise setting an initial state for the widget within the context. Certain of these embodiments further comprise managing selection internally at the widget. In some of these embodiments, managing selection includes maintaining selection state and data binding for the widget. Certain of these embodiments further comprise modifying the selection state in response to an event. In some of these embodiments, the modifying includes causing the widget to query the centralized context. In some of these embodiments, the modifying includes updating the centralized context. In some of these embodiments, the modifying includes updating the associated data storage. In some of these embodiments, the modifying includes receiving updated data from the associated data storage at the widget by querying a data access interface. In some of these embodiments, the modifying includes causing the data access interface to query the centralized context. In some of these embodiments, the modifying includes providing data to the widget based on a response to the query of the data access interface.
Certain embodiments of the invention provide a method for capturing file formats. In some of these embodiments, the line-by-line determination of a file format can be modified by automatically capturing file format information directly from the file input/output methods in the source code. In some of these embodiments, the file format specification is automatically derived from source code. In some of these embodiments, write operations in source code are instrumented such that the file format specification can be captured by a “dummy” file capturing object and subsequently processed to produce the file structure specification. In some of these embodiments, a layered input/output mechanism for working with files is provided.
In some of these embodiments, a manual examination of the source code is used to find a first write operation. In some of these embodiments, sequential write operations are tracked and recorded. In some of these embodiments, an automatic line-by-line determination is performed. In some of these embodiments, source code write operations are instrumented with descriptions and/or a mechanism to detect data type and size. In some of these embodiments, instrumented source code is compiled into a binary executable form 212. In some of these embodiments, the instrumented source code writes to a file capturing object. In some of these embodiments, the capturing object automatically detects when a write operation occurs and identifies metadata (data type, size, and/or description) associated with the write operation. In some of these embodiments, a file format specification is produced.
Certain embodiments of the invention provide methods for automatically capturing file format information directly from the file input/output, comprising instrumenting certain file operations to capture file format information, collecting the file format information in a captured file object and processing the captured file object to obtain the file structure specification.
Although the present invention has been described with reference to specific exemplary embodiments, it will be evident to one of ordinary skill in the art that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.
The present application claims priority from U.S. Provisional Patent Application No. 60/985,962 filed Nov. 6, 2007, entitled “Centralized Selection Context for User Interface Data Binding and Event Handling,” which is incorporated by reference herein in its entirety.
Number | Date | Country | |
---|---|---|---|
60985962 | Nov 2007 | US |